[Upgrade] Rollback of 5.1.1 upgrade wasn't done right - ostf and fuelclient failed with authorization failure

[Andrey Sledzinskiy]

{

    "build_id": "2014-09-17_21-40-34",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "11",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
    "production": "docker",
    "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
    "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "release_versions": {
        "2014.1.1-5.1": {
            "VERSION": {
                "build_id": "2014-09-17_21-40-34",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "11",
                "api": "1.0",
                "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
                "production": "docker",
                "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
                "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1",
                "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"
            }
        }
    },
    "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"

}

Upgrade script version
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1.1"
  api: "1.0"
  build_number: "3"
  build_id: "2014-11-12_21-56-33"
  astute_sha: "702af3db6f5bca92525bc8322d7d5d7675ec857e"
  fuellib_sha: "7d711bb3d2572d55463cd38b9df3ad7a20a34262"
  ostf_sha: "64cb59c681658a7a55cc2c09d079072a41beb346"
  nailgun_sha: "bbc9dfe78a0c33040dcd16de9a40a3491788719c"
  fuelmain_sha: "e5e534ade6f3765a87feee3d44d39df68ae28f80"

Steps:
1. Deploy new 5.1.1 env - Simple, Neutron Gre, CentOS, 1 controller, 2 compute+cinder
2. Modify 5.1.1 upgrade script to fail
3. Run upgrade

Expected - rollback was successfull, everything is operational
Actual - rollback passed, but there are problems with authorization in fuel client and ostf server is unavailable

[root@nailgun ~]# fuel nodes
Traceback (most recent call last):
  File "/usr/bin/fuel", line 9, in <module>
    load_entry_point('fuelclient==0.2', 'console_scripts', 'fuel')()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 105, in wrapper
    handle_exceptions(exc)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 177, in main
    parser.parse()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 93, in parse
    actions[parsed_params.action].action_func(parsed_params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 58, in action_func
    method(params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/node.py", line 243, in list
    node_collection = NodeCollection.get_all()
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/node.py", line 152, in get_all
    return cls(Node.get_all())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 68, in get_all
    return map(cls.init_with_data, cls.get_all_data())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 64, in get_all_data
    return cls.connection.get_request(cls.class_api_path)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 151, in get_request
    request.add_header('X-Auth-Token', self.auth_token)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 67, in auth_token
    if self.auth_required:
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 76, in auth_required
    request = urllib2.urlopen(''.join([self.api_root, 'version']))
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine

But after some period of time fuel client started working

Logs are attached

[Ihor Kalnytskyi (ikalnytskyi)]

Very strange and not enough info. It looks like there's some delay between fuel_upgrade script is done and all services are running. Need to investigate, but it's definitely not a Critical, because some timeout (1min) is enough to avoid this issue.

[Andrey Sledzinskiy (asledzinskiy)]

Need to modify test to wait until all services are ready

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/135310

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135310
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=a6d5f5ae9a9c2a0adaed527b729eb0cab6c2a2d5
Submitter: Jenkins
Branch: stable/5.1

commit a6d5f5ae9a9c2a0adaed527b729eb0cab6c2a2d5
Author: asledzinskiy <email address hidden>
Date: Tue Nov 18 16:37:18 2014 +0200

    Add wait until nailgun will be operable to rollback tests

    -In case of rollback some services need time to start
    so we need to wait in tests for these services to start

    Change-Id: I34ae6388fe023cad929c1c039d4f7643658548be
    Closes-Bug: #1392393

[Mike Scherbakov (mihgen)]

Colleagues,
I see a big issue with this bug report. It doesn't target 5.1.1, but fix was merged to stable/5.1. More over, there was no patch to master first, so it's in violation of https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Backport_bugfixes_to_stable_release_series. Please fix.

[Andrey Sledzinskiy (asledzinskiy)]

Sorry , I put wrong closes-bug in commit to master - it's already merged to master
https://review.openstack.org/#/c/136265/