standalone-upgrade failing during deploy steps for CalledProcessError: Command '['systemctl', 'stop', u'tripleo_keystone.service']' returned non-zero exit status

[Marios Andreou]

at [1][2] the tripleo-ci-centos-7-standalone-upgrade fails during the deploy steps "TASK [Start containers for step 3]" with trace like:

 2019-04-22 08:40:22 | 2019-04-22 08:40:22.948 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "Traceback (most recent call last):",
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.948 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] " File \"/usr/lib/python2.7/site-packages/paunch/utils/systemd.py\", line 123, in service_delete",
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.949 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] " subprocess.check_call(['systemctl', 'stop', sysd_f])",
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.949 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] " File \"/usr/lib64/python2.7/subprocess.py\", line 542, in check_call",
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.949 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] " raise CalledProcessError(retcode, cmd)",
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.950 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "CalledProcessError: Command '['systemctl', 'stop', u'tripleo_keystone.service']' returned non-zero exit status 1",
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.950 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "Command '['systemctl', 'stop', u'tripleo_keystone.service']' returned non-zero exit status 1"
 2019-04-22 08:40:22 | 2019-04-22 08:40:22.951 102903 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] ]

Looks like the error is coming from paunch at [3]

[1] http://logs.openstack.org/89/651789/6/check/tripleo-ci-centos-7-standalone-upgrade/c6bfa88/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-04-22_08_40_22
[2] http://logs.openstack.org/20/652920/4/check/tripleo-ci-centos-7-standalone-upgrade/4c94f22/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz
[3] https://github.com/openstack/paunch/blob/37d6058b60ab777b58e0413e4ab8ed1422c54b12/paunch/utils/systemd.py#L122-L127

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/654952

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.opendev.org/654952

[Marios Andreou (marios-b)]

I came here thinking i could close this as I'd seen some green runs for the standalone-upgrade on master. However looking at the top two fails in [1] at time of writing i found [2] and [3] which have the same signature but for different services - so this is likely no keystone specific:

    (from [2]): 2019-05-16 00:40:19 | 2019-05-16 00:40:19.727 104318 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "CalledProcessError: Command '['systemctl', 'stop', u'tripleo_nova_compute.service']' returned non-zero exit status 1",

    (from [3]): 2019-05-15 23:53:11 | 2019-05-15 23:53:11.597 95831 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "CalledProcessError: Command '['systemctl', 'stop', u'tripleo_cinder_volume.service']' returned non-zero exit status 1",

However I am lowering the priority for this. The job is non voting and expected to be volatile which is why master is non voting but stein is voting for standalone-upgrade.

[1] http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade
[2] http://logs.openstack.org/94/659394/1/check/tripleo-ci-centos-7-standalone-upgrade/1db3a41/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-05-16_00_40_19
[3] http://logs.openstack.org/58/658358/5/check/tripleo-ci-centos-7-standalone-upgrade/b98b140/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

[Marios Andreou (marios-b)]

lowered to High from Critical and removed alert tag please re-add if you disagree