[Upgrade] Upgrade of fuel failed on upgrading openstack stage withHTTPError: 401 Client Error: Unauthorized

[Andrey Sledzinskiy]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "55"
  build_id: "2014-12-17_16-14-11"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "73332192a257ea02c40a39885c502ad1ebdf3eda"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "dc621c96e30f0dd284998e975f3b800afe9fc082"
  fuelmain_sha: "45caacadb878abfbd9d60e134d72229698b469c9"

Steps:
1. Deploy next 5.1.1-48 cluster - CentOS, simple, Nova-network flat, ceph for volumes and for images, 1 controller+ceph, 2 compute+ceph
2. Upload upgrade script to fuel master
3. Run upgrade

Actual result - upgrade failed with

2014-12-18 05:18:06 ERROR 26594 (upgrade) *** UPGRADE FAILED
2014-12-18 05:18:06 ERROR 26594 (cli) 401 Client Error: Unauthorized
Traceback (most recent call last):
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 177, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 169, in run_upgrade
    upgrade_manager.run()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 51, in run
    upgrader.upgrade()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/openstack.py", line 61, in upgrade
    self.install_releases()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/openstack.py", line 144, in install_releases
    existing_releases = self.nailgun.get_releases()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/clients/nailgun_client.py", line 49, in get_releases
    r.raise_for_status()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 401 Client Error: Unauthorized

This issue wasn't reproduced during manual testing of upgrades but half of upgrade tests on CI failed with this error today

[Evgeniy L (rustyrobot)]

Discussed with Igor, it's not critical because it's reproducible only on CI, was decided to add retries for authorization.

[Ihor Kalnytskyi (ikalnytskyi)]

It looks like we have next scenario:

1. Keystone service has been started by puppet - https://github.com/stackforge/fuel-main/blob/master/docker/keystone/start.sh#L9
2. The fuel upgrade checkers were passed successfully, and the OpenStack upgrader is about to execute.
3. The keystone service is shutdown due to https://github.com/stackforge/fuel-main/blob/master/docker/keystone/start.sh#L15
4. OpenStack upgrader fails to retrieve token.

Solution?

We should either do not restart keystone or do not launch it by puppet.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140016
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=81e01c06420db7289f1efaaf8eddf172b61dca45
Submitter: Jenkins
Branch: master

commit 81e01c06420db7289f1efaaf8eddf172b61dca45
Author: Igor Kalnitsky <email address hidden>
Date: Mon Dec 8 16:19:09 2014 +0200

    fuel_upgrade: add retries for getting token

    After starting a new Keystone container, the first attempt to get a token
    fails. Unfortunately, the logs keep silent and say nothing. As a workaround,
    we can just increase retries number.

    Change-Id: Ib1bba120614d1964e838c289489f1877074c324d
    Closes-Bug: #1403798

[Tatyanka (tatyana-leontovich)]

reproduced here http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0.system_test.CentOS.ceph_multinode_compact.upgrade/40/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/156574

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156574
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=54b742f054e464c285aa1929c630b0d25a876186
Submitter: Jenkins
Branch: stable/6.0

commit 54b742f054e464c285aa1929c630b0d25a876186
Author: Igor Kalnitsky <email address hidden>
Date: Mon Dec 8 16:19:09 2014 +0200

    fuel_upgrade: add retries for getting token

    After starting a new Keystone container, the first attempt to get a token
    fails. Unfortunately, the logs keep silent and say nothing. As a
    workaround, we can just increase retries number.

    (cherry picked from commit 81e01c06420db7289f1efaaf8eddf172b61dca45)

    Change-Id: Ib1bba120614d1964e838c289489f1877074c324d
    Closes-Bug: #1403798