Unexpected dockerctl lock after deployment

[Egor Kotko]

The CI test backup_restore_master_base failed:
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.backup_restore_master/35/testReport/%28root%29/backup_restore_master_base/backup_restore_master_base/

        Scenario:
            1. Revert snapshot "empty"
            2. Backup master
            3. Check backup
            4. Restore master
            5. Check restore
            6. Check iptables

Actual result:

+ set -e
+ lock
+ [[ '' -eq 1 ]]
+ exec
+ flock -n 904
+ echo 'Failed to acquire the lock.'
+ exit 1

[Bogdan Dobrelya (bogdando)]

Raising to critical as the locking errors are blocker for the master node containers.
For my env deployed from the build #233, all containers but nginx shown in the BACKOFF state. And I cannot neither restart them with nailgun (ERROR (already started)) nor destroy and start with dockerctl ("Failed to acquire the lock")

[Bogdan Dobrelya (bogdando)]

I found a w/a. The command removing the lock all the time, made all containers to be run by supervisorctl logic, eventually.

while true; do rm -f /var/lock/dockerctl ; sleep 1; done &

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/#/c/242488/ fixes this (still not merged)

[Vasily Gorin (vgorin)]

Verified by CI: https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.backup_restore_master/52/testReport/(root)/backup_restore_master_base/

[Vasily Gorin (vgorin)]

Verified by CI on ISO 207