fuel master upgrade from 6.0 to 6.1 failed

[Sergey Murashov]

i used fuel-6.1-upgrade-521-2015-06-08_06-13-27.tar.lrz
Steps to reproduce:
1)Install fuel master 6.0
2)Download upgrade tar.lrz to 6.1
3)lrzuntar it
4) run upgrade.sh

Actual result:
It tries to call keystone, but keystone is unavailable. Part of the log:
2015-06-09 13:41:31 DEBUG 9642 (health_checker) Start integration_postgres_nailgun_nginx checker
2015-06-09 13:41:34 DEBUG 9642 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 113] No route to host)
2015-06-09 13:41:37 DEBUG 9642 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 113] No route to host)
2015-06-09 13:41:40 DEBUG 9642 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 113] No route to host)
2015-06-09 13:41:43 DEBUG 9642 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 113] No route to host)
2015-06-09 13:41:43 ERROR 9642 (keystone_client) Cannot retrieve an auth token - an unauthenticated request will be performed.
Traceback (most recent call last):
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/clients/keystone_client.py", line 61, in request
    token = utils.wait_for_true(self.get_token, timeout=10)
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 206, in wait_for_true
    'command with timeout {0}'.format(timeout))
TimeoutError: Failed to execute command with timeout 10

And at the end we get:
2015-06-09 13:56:10 ERROR 9642 (upgrade) *** UPGRADE FAILED
2015-06-09 13:56:10 ERROR 9642 (cli) Failed to run services ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'integration_nginx_nailgun', 'ostf', 'rabbitmq', 'nginx', 'keystone', 'mcollective', 'cobbler', 'integration_rabbitmq_astute_nailgun']
Traceback (most recent call last):
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 176, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 168, in run_upgrade
    upgrade_manager.run()
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 80, in run
    upgrader.upgrade()
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 88, in upgrade
    self.upgrade_verifier.verify()
  File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/health_checker.py", line 403, in verify
    self._get_non_running_services()))
UpgradeVerificationError: Failed to run services ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'integration_nginx_nailgun', 'ostf', 'rabbitmq', 'nginx', 'keystone', 'mcollective', 'cobbler', 'integration_rabbitmq_astute_nailgun']
==================================================

Couldn't start some of the services, try to run upgrade again.

Upgrade failed

[Ihor Kalnytskyi (ikalnytskyi)]

1/ The upgrade wasn't run from fresh installed 6.0. It was a double upgrade: 5.1.1 -> 6.0 -> 6.1

2/ It's hard to say why upgrade procedure failed to retrieve keystone token. There may be a plenty of issues, I think it's something floating and second run (upgrade) should work fine.

3/ Unfortunately, rollback procedure failed too due to one of Docker issues with stale mounted resources. We can't fix it - it's up to Docker.

Move it to Incomplete for now. Let's try to reproduce it once again.

[Sergey Murashov (smurashov)]

I've reproduced it again,
I installed fuel 5.1.1, deploy OS cluster(HA, all ceph, neutron gre)
Upgrade from 5.1.1 to 6.0.
Upgrade from 6.0 to 6.1.
And i got the same error

[Ihor Kalnytskyi (ikalnytskyi)]

By some reason, new docker-io package wasn't installed and new docker images aren't compatible with old docker.

Continue investigation.

[Andrey Sledzinskiy (asledzinskiy)]

I couldn't reproduce it on my env
Steps:
1. Install 5.1.1-48 fuel and deploy cluster with Neutron gre, HA, ceph
2. Upgrade it to 6.0-58 version
3. Upgrade fuel to 6.1-521 version

And it all passed.
Sergey, may be you did some specific actions cause it seems that simple chain of upgrades works fine.

[Ihor Kalnytskyi (ikalnytskyi)]

@Sergey, could you please try again and contact me when you're ready to run upgrade to 6.1?

p.s: move it to incomplete until then

[Sergey Murashov (smurashov)]

The reason was in wrong locale

[Sergey Murashov (smurashov)]

i've found https://tickets.puppetlabs.com/browse/PUP-736, looks like we have package which have description not in UTF-8 encoding

[Ihor Kalnytskyi (ikalnytskyi)]

@Sergey,

Nope, we have a package which description isn't ASCII-compatible. Even UTF-8 may break your execution, because `yum` was fallbacked to US-ASCII.