UMM doesn't respect "COUNTER_RESET_TIME" value

[Alexander Rubtsov]

 --- Environment ---
Mirantis OpenStack: 9.0 + code from stable/mitaka branch on January 22
Environment: All OpenStack nodes are VMs on top of Libvirt+KVM

 --- Description ---
A node ends up in maintenance mode (UMM) after the configured number of "unclean" reboots ("REBOOT_COUNT" in umm.conf) even if the period of time between these reboots is longer than the configured one (COUNTER_RESET_TIME in umm.conf).

 --- Steps to reproduce ---
1) Deploy a MOS environment on top of Libvirt+KVM

2) Ensure there are default values of "REBOOT_COUNT" and "COUNTER_RESET_TIME" is configured on a Controller node
cat /etc/umm.conf
UMM=yes
REBOOT_COUNT=2
COUNTER_RESET_TIME=10

3) Kill a Libvirt VM which corresponds to Controller node using virsh then start it immediately
virsh destroy f90-reg-node-4-controller
virsh start f90-reg-node-4-controller

4) Enter the Controller node and check that the operating system has booted properly
$ umm status
runlevel N 2

5) Wait for 15 minutes

6) Kill the same Libvirt VM using virsh then start it immediately
virsh destroy f90-reg-node-4-controller
virsh start f90-reg-node-4-controller

 --- Actual behavior ---
The node is in maintenance mode
# umm status
umm

 --- Expected behavior ---
Operating system is still boot in runlevel 2
$ umm status
runlevel N 2

[Alexander Rubtsov (arubtsov)]

sla1 for 9.0-updates

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/436906

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/436906
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3841653603787d97e2ca3e7466ade6e21784bd5e
Submitter: Jenkins
Branch: stable/mitaka

commit 3841653603787d97e2ca3e7466ade6e21784bd5e
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Feb 22 14:27:49 2017 +0200

    Enable umm time reset script on boot

    Change-Id: I73cafc1bd4e0abf46d281f6139f99fd8439af006
    Closes-Bug: 1665669

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/441906

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/441907

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/441906
Reason: wrong commit

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 + mu1 updates.

Steps to verify:
scenario 1:
1) reboot of controller (sudo reboot)
2) ssh to controller, check UMM_DRC value
3) virsh destroy fuel_custom_slave-01
   virsh start fuel_custom_slave-01
4) ssh to controller, check UMM_DRC value, umm status
5) waiting for 15 min
6) check UMM_DRC value
7) virsh destroy fuel_custom_slave-01
   virsh start fuel_custom_slave-01
8) ssh to controller, check UMM_DRC value, umm status

scenario 2:
1) reboot of controller (sudo reboot)
2) ssh to controller, check UMM_DRC value
3) waiting for 15 min
4) ssh to controller, check UMM_DRC value
5) virsh destroy fuel_custom_slave-01
   virsh start fuel_custom_slave-01
6) ssh to controller, check UMM_DRC value, umm status
7) waiting for 15 min
8) check UMM_DRC value
9) virsh destroy fuel_custom_slave-01
   virsh start fuel_custom_slave-01
10) ssh to controller, check UMM_DRC value, umm status

Actual results after the fix:
scenario 1:
2) root@node-1:~# cat /var/lib/umm/*
   UMM_DRC=1
4) root@node-1:~# cat /var/lib/umm/*
   UMM_DRC=2
   root@node-1:~# umm status
   runlevel N 2
6) root@node-1:~# cat /var/lib/umm/*
   cat: /var/lib/umm/*: No such file or directory
8) root@node-1:~# cat /var/lib/umm/*
   UMM_DRC=1
   root@node-1:~# umm status
   runlevel N 2

scenario 2:
2) root@node-1:~# cat /var/lib/umm/*
   UMM_DRC=1
4) root@node-1:~# cat /var/lib/umm/*
   cat: /var/lib/umm/*: No such file or directory
6) root@node-1:~# cat /var/lib/umm/*
   UMM_DRC=1
   root@node-1:~# umm status
   runlevel N 2
8) root@node-1:~# cat /var/lib/umm/*
   cat: /var/lib/umm/*: No such file or directory
10)root@node-1:~# cat /var/lib/umm/*
   UMM_DRC=1
   root@node-1:~# umm status
   runlevel N 2

Sometimes UMM_DRC counter doesn't increase value after virsh destroy/start - separate bug was created to track this issue - https://bugs.launchpad.net/fuel/+bug/1671827.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/451281

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/451281
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/441907
Reason: This repo is retired now, no further work will get merged.