[astute] For some reason astute erased node before reinstall with partition preservation and caused data loss

[Roman Sokolkov]

Astute has a bug causes unexpected node erasure.

We are doing upgrade from MOS7.0 to MOS9.1 and using partition presevation.

Two of our nodes failed to reprovision and immediately were erased by astute.

Which caused data loss. At least we lost /var/lib/nova XFS partition.

Initial root cause is that astute has only 4 minutes timeout for node reboot. But hardware node takes ~10min usually.

2016-10-17 12:15:18 WARNING [1041] Time detection (240 sec) for node reboot has expired
2016-10-17 12:15:18 WARNING [1041] Reboot command failed for nodes ["10"]. Check debug output for details
2016-10-17 12:15:18 DEBUG [1041] Task time summary: reboot_reprovisioned_nodes with status error on node 10 took 00:05:00

Step-by-step:
1) Astute reboots node and gaves up waitning (4min timeout)
2) Step [1] which writes systemtype "image" fails (because node still offline)
3) This later causes code block [2] to be executed after provisioning failed (not related issue).

Finally node erased. See log [3] :(

1 - https://github.com/openstack/fuel-astute/blob/stable/mitaka/lib/astute/provision.rb#L238
2 - https://github.com/openstack/fuel-astute/blob/stable/mitaka/lib/astute/provision.rb#L136-L140
3 - http://paste.openstack.org/show/586292/

[Ilya Kharin (akscram)]

I would also mention here that this problem is not only about timeouts but in the first place this is about robustness of the partitions preservation mechanism. The keep_data=True flag which can be set for partitions have to mean that any operations which can be lead to potential data loss have to be excluded for disks with such partitions.

[Ilya Kharin (akscram)]

For for anyone interested in the history of the appearance of this behaviour, please take a look on the next bugfix:
    https://review.openstack.org/136728

[Evgeniy L (rustyrobot)]

Ilya,

Upgrade procedure must never use disks preservation feature, it's broken by design, since there are some cases when it may work, and dozens of cases when it breaks the system (with possible data corruption), I'm pretty confused to see why we even started to do it this way for upgrades, why do we even need do full re-provisioning? It's much simpler and much much less error-prone to download new image and write it into OS volume, that is it, no re-partitioning is required.

I'm not sure if description of the ticket is correct:

1. Astute always erases (MBR) of disks when you ask to re-provision the node.

2. The patch you are referring to is irrelevant (see previous item). Re-erasing of MBR multiple times does not make it worse.

3. Looking at the environment it became pretty clear, that the problem was that superblock was erased and partitioning schema didn't match the one, which was previously used, it was pretty obvious by running xfs_repair which failed to restore superblock from secondary blocks, due to differentiation in offset. As result what happened is you asked to re-partition the system with the new schema (new disk sizes, due to new iscsi volume found), even if some of volumes have keep_data flag set, re-partitioning of other volumes (w/o keep_data) pretty easily can break something (e.g. it may easily overlap with some raw partition).

My suggestion would be to implement proper and explicit rollout of new operating system, without dances with setting dozens of flags in dozens of places and praying that it will eventually work, and there are no differences in kernel and fs-related tooling, which can pretty easily break something.

[Ilya Kharin (akscram)]

@Evgeny, could you please take a look on links that Roman mentioned? Especially on [3] and then on [2] it must explain something.

[Evgeniy L (rustyrobot)]

@Ilya, I've debugged the env with Roman once more, it appeared to be that sometimes upgrade succeeds due to a bug in MCagent which should always erase disks, in some cases it doesn't do it properly. Still usage of `keep_data=True` for upgrade procedure is a dangerous operation to do, and broken by design.

[Vladimir Kozhukalov (kozhukalov)]

btw, now reboot_timeout is set to 900 seconds in master and 600 seconds in stable/mitaka (i have no idea why they are different, btw)

[1] https://github.com/openstack/fuel-astute/blob/master/lib/astute/config.rb#L71
[2] https://github.com/openstack/fuel-astute/blob/stable/mitaka/lib/astute/config.rb#L71

[Vladimir Kozhukalov (kozhukalov)]

Fuel-agent allows you to run tasks granularly (only partitioning, only copying image, etc.) [1] Given that and also taking into account provision-as-a-graph feature one can potentially create granular tasks for upgrade graph. If you need partitions to stay as they are, you can use tasks that only copy a new image, etc. Closing this bug as 'won't fix'.

[1] https://github.com/openstack/fuel-agent/blob/master/setup.cfg#L18-L25