[OSTF] Update image over v1 failed with 400 bad requestAuthentication failed during image update: Authentication required

Bug #1461036 reported by Tatyanka
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Alexey Galkin
7.0.x
Invalid
Medium
Alexey Galkin

Bug Description

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "492"
  build_id: "2015-06-01_10-17-59"
  nailgun_sha: "3830bdcb28ec050eed399fe782cc3dd5fbf31bde"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "5d570ae5e03909182db8e284fbe6e4468c0a4e3e"
  fuel-library_sha: "c9a86ac0e6da95d36e328ce5130715792a2eb177"
  fuel-ostf_sha: "7413186490e8d651b8837b9eee75efa53f5e230b"
  fuelmain_sha: "6b5712a7197672d588801a1816f56f321cbceebd"

Steps:
"""Deploy ceph with cinder in HA mode

        Scenario:
            1. Create cluster
            2. Add 3 nodes with controller and ceph OSD roles
            3. Add 1 node with ceph OSD roles
            4. Add 2 nodes with compute and ceph OSD roles
            5. Deploy the cluster
            6. Revert snapshot with ceph cluster in HA mode
            7. Check ceph status
            8. RUN OSTF

Actual result:
ostf fail with:
urllib3.connectionpool: DEBUG: "POST http://10.109.1.2:5000/v2.0/tokens HTTP/1.1" 200 3762
iso8601.iso8601: DEBUG: Parsed 2015-06-02T05:23:25Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'05', 'daydash': u'02', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'25', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'06', 'day': None, 'minute': u'23'} with default timezone <iso8601.iso8601.Utc object at 0x2e0b650>
iso8601.iso8601: DEBUG: Got u'2015' for 'year' with default None
iso8601.iso8601: DEBUG: Got u'06' for 'monthdash' with default None
iso8601.iso8601: DEBUG: Got 6 for 'month' with default 6
iso8601.iso8601: DEBUG: Got u'02' for 'daydash' with default None
iso8601.iso8601: DEBUG: Got 2 for 'day' with default 2
iso8601.iso8601: DEBUG: Got u'05' for 'hour' with default None
iso8601.iso8601: DEBUG: Got u'23' for 'minute' with default None
iso8601.iso8601: DEBUG: Got u'25' for 'second' with default None
keystoneclient.auth.identity.v2: DEBUG: Making authentication request to http://10.109.1.2:5000/v2.0/tokens
urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.109.0.4
urllib3.connectionpool: DEBUG: Setting read timeout to None
urllib3.connectionpool: DEBUG: "POST http://10.109.1.2:5000/v2.0/tokens HTTP/1.1" 200 3762
glanceclient.common.http: DEBUG: curl -i -X POST -H 'Accept-Encoding: gzip, deflate, compress' -H 'x-image-meta-container_format: bare' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}eb49a8536529f3e964d5f6c2ac98bd75bfe70875' -H 'x-image-meta-size: 1024' -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: raw' -H 'x-image-meta-name: ostf_test-image_glance-657521942' http://10.109.1.2:9292/v1/images
urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.109.0.4
glanceclient.common.http: DEBUG: Request returned failure status 400.
urllib3.connectionpool: WARNING: HttpConnectionPool is full, discarding connection: 10.109.0.4
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 176, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/glancemanager.py", line 65, in image_create
    disk_format=disk_format, **kwargs)
  File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 288, in create
    data=image_data)
  File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 262, in post
    return self._request('POST', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 227, in _request
    raise exc.from_response(resp, resp.content)
HTTPBadRequest: <html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  Authentication failed during image update: Authentication required<br /><br />

 </body>
</html> (HTTP 400)

--------------------- >> end captured

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Download full text (3.3 KiB)

The same problem with glance, when we tried to get image list after l3 failover on one of the node

Scenario:
Check l3-agent rescheduling after reset non-primary controller

        Scenario:
            1. Revert snapshot with neutron cluster
            2. Create an instance with a key pair
            3. Manually reschedule router from primary controller
               to another one
            4. Reset controller with l3-agent
            5. Check l3-agent was rescheduled
            6. Check network connectivity from instance via
               dhcp namespace
            7. Run OSTF

On step 6 when we retrieve image list we recieve 400 bad error:

ERROR: Check l3-agent rescheduling after reset non-primary controller
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron_destructive/fuelweb_test/helpers/decorators.py", line 70, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_neutron.py", line 239, in neutron_l3_migration_after_reset
    os_conn, remote).addresses['net04'][0]['addr']
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron_destructive/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_neutron.py", line 53, in create_instance_with_keypair
    neutron=True, key_name='instancekey')
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron_destructive/fuelweb_test/helpers/os_actions.py", line 79, in create_server_for_migration
    image_id = self._get_cirros_image().id
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron_destructive/fuelweb_test/helpers/os_actions.py", line 37, in _get_cirros_image
    for image in self.glance.images.list():
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/glanceclient/v1/images.py", line 249, in list
    for image in paginate(params, return_request_id):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/glanceclient/v1/images.py", line 233, in paginate
    images, resp = self._list(url, "images")
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list
    resp, body = self.client.get(url)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/glanceclient/common/http.py", line 262, in get
    return self._request('GET', url, **kwargs)
  F...

Read more...

Changed in fuel:
assignee: MOS QA Team (mos-qa) → MOS Keystone (mos-keystone)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
assignee: MOS Keystone (mos-keystone) → Boris Bobrov (bbobrov)
Changed in fuel:
importance: Undecided → High
tags: added: keystone
Changed in fuel:
status: New → Confirmed
Changed in fuel:
assignee: Boris Bobrov (bbobrov) → Alexander Makarov (amakarov)
Revision history for this message
Mike Fedosin (mfedosin) wrote :

I have a question about second case (check l3-agent rescheduling after reset non-primary controller). As far as I see master glanceclient version is used there. It's not compatible with MOS 6.1, only 0.14.0 - 0.15.0 is accepted. So, what exactly do you test there?

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

As about scenario added in comment #2, separate issue is created to qa team https://bugs.launchpad.net/fuel/+bug/1461173. So we can ignore comment

description: updated
Revision history for this message
Boris Bobrov (bbobrov) wrote :

Waiting for our QA teams to reproduce it without step "Revert snapshot with ceph cluster in HA mode".

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Aleksander, do you have clear desciption - yes, do you have diagnostic snapshot - yes. Please, go forward to find root cause.

Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

I will try to reproduce the issue on my environment and will update the status. Not it is changed to Incomplete because I'm not sure that it will be reproduced (but of course we need to verify it)

Changed in fuel:
assignee: Alexander Makarov (amakarov) → Timur Nurlygayanov (tnurlygayanov)
status: Confirmed → Incomplete
Alexey Galkin (agalkin)
Changed in fuel:
assignee: Timur Nurlygayanov (tnurlygayanov) → Aleksey (agalkin)
Changed in fuel:
assignee: Alexey Galkin (agalkin) → MOS QA Team (mos-qa)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

I can't reproduce this issue on my test environment, looks like the root of this problem in snapshots revert or some issues with sync after the destructive scenarios. We need to try to reproduce the issue with snapshot to get the root of the problem.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/keystone (openstack-ci/fuel-6.1/2014.2)

Related fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/7504

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to openstack/keystone (openstack-ci/fuel-6.1/2014.2)

Reviewed: https://review.fuel-infra.org/7504
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: f838acf2a38830829c479ab3556d442b6953abac
Author: Alexander Makarov <email address hidden>
Date: Thu Jun 4 15:48:18 2015

Try all available servers on memcache get()

In case of the cache miss try all other servers in the bucket

Change-Id: I59c572b1c572ef5b81670837ec11daa3d1137e14
Related-Bug: 1461036

Changed in fuel:
status: Incomplete → Fix Committed
Revision history for this message
Egor Kotko (ykotko) wrote :

{"build_id": "2015-06-08_06-13-27", "build_number": "521", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-08_06-13-27", "build_number": "521", "api": "1.0", "fuel-library_sha": "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d", "nailgun_sha": "4340d55c19029394cd5610b0e0f56d6cb8cb661b", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "7766818f079881e2dbeedb34e1f67e517ed7d479", "fuel-ostf_sha": "7c938648a246e0311d05e2372ff43ef1eb2e2761", "release": "6.1", "fuelmain_sha": "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d", "nailgun_sha": "4340d55c19029394cd5610b0e0f56d6cb8cb661b", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "7766818f079881e2dbeedb34e1f67e517ed7d479", "fuel-ostf_sha": "7c938648a246e0311d05e2372ff43ef1eb2e2761", "release": "6.1", "fuelmain_sha": "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"}

http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.thread_3/151/testReport/%28root%29/check_ceph_ha/check_ceph_ha/

http://paste.openstack.org/show/277137/

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Egor Kotko (ykotko) wrote :

glance-api.log on the first controller contains problem with token:
http://paste.openstack.org/show/277210/

Revision history for this message
Egor Kotko (ykotko) wrote :
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Ok, I will test this issue on my environment with RC1 ISO one more time.

Changed in fuel:
assignee: MOS QA Team (mos-qa) → Timur Nurlygayanov (tnurlygayanov)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Script which help to detect some issues with keystone during the destructive tests (power on and power off different controllers):

for i in `seq 1 1000`; do
  echo `date`
  AUTH_URL='http://172.16.0.2:5000/v2.0/'
  test_token=`OS_TENANT_NAME='admin' OS_USERNAME='admin' OS_PASSWORD='admin' OS_AUTH_URL=$AUTH_URL keystone token-get | awk '{print $4}' | sed '5q;d'`
  for i in `seq 0 9`; do
      keystone --os-token=$test_token --os-endpoint=$AUTH_URL tenant-list 1> /dev/null
  done
done

if we will set dead_retry=30 we can see that downtime of keystone service with token validation less than with default parameters:
http://paste.mosi.mirantis.net/show/554/

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

with default value of parameter we can see that we have long downtime:

http://paste.mosi.mirantis.net/show/555/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190248

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/190249

tags: added: 6.1rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/190248
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8f461c226471bd6dc6122387b92f0fc226465773
Submitter: Jenkins
Branch: master

commit 8f461c226471bd6dc6122387b92f0fc226465773
Author: Timur Nurlygayanov <email address hidden>
Date: Wed Jun 10 18:48:09 2015 +0300

    Fixed issue with keystone memcache

    With the default value for the 'dead_timeout' recovery
    of keystone memcache service can take more than 3 minutes.
    This parameter allows to reduce this timeout to 20-30 seconds.

    Change-Id: Iac5076532e25295f6d9abbb9e9c1b4213931d1da
    Closes-Bug: #1461036

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/6.1)

Reviewed: https://review.openstack.org/190249
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bf1382cfacc85bfc58cb49f98ce4ce13fa01d651
Submitter: Jenkins
Branch: stable/6.1

commit bf1382cfacc85bfc58cb49f98ce4ce13fa01d651
Author: Timur Nurlygayanov <email address hidden>
Date: Wed Jun 10 19:25:04 2015 +0300

    Fixed issue with keystone memcache

    With the default value for the 'dead_timeout' recovery
    of keystone memcache service can take more than 3 minutes.
    This parameter allows to reduce this timeout to 20-30 seconds.

    Change-Id: Iac5076532e25295f6d9abbb9e9c1b4213931d1da
    Closes-Bug: #1461036

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/keystone (openstack-ci/fuel-7.0/2015.1.0)

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/8189

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/keystone (openstack-ci/fuel-7.0/2015.1.0)

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8189
Reason: Not needed in 7.0

Revision history for this message
Egor Kotko (ykotko) wrote :

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "26"
  build_id: "2015-07-06_18-08-24"
  nailgun_sha: "d040c5cebc9cdd24ef20cb7ecf0a337039baddec"
  python-fuelclient_sha: "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45"
  astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
  fuel-library_sha: "251c54e8de2f41aacd260751e7a891e9fbffc45d"
  fuel-ostf_sha: "a752c857deafd2629baf646b1b3188f02ff38084"
  fuelmain_sha: "4f2dff3bdc327858fa45bcc2853cfbceae68a40c"

http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.centos.ha_neutron_destructive/18/testReport/%28root%29/ha_neutron_test_3_1_rabbit_failover/ha_neutron_test_3_1_rabbit_failover/

http://paste.openstack.org/show/352097/

Revision history for this message
Egor Kotko (ykotko) wrote :
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

affected kilo failover now as well:
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.ha_neutron_destructive/19/testReport/junit/(root)/neutron_l3_migration_after_reset/neutron_l3_migration_after_reset/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "239"
  build_id: "2015-07-07_18-02-36"
  nailgun_sha: "2f8d9be6bca0c32a83acda02e1689a44f0d8eefe"
  python-fuelclient_sha: "018c53561baa87397e50368c7f48674829c322c0"
  astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
  fuel-library_sha: "93a5fe7251f3eb932d588ba338920e9e7cfe7f0c"
  fuel-ostf_sha: "ba48d52474515bcd1ad2a9dfa5228bff36ee98d7"
  fuelmain_sha: "c53bb47881f0bb9ba14274a596923d9860140406"

tags: added: kilo
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

From ostf.log :

"HttpConnectionPool is full, discarding connection: .." when glance client sends an authorization request.
Looks like it is caused by this class requests.adapters.HTTPAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False) , when pool_maxsize is reached .

HTTPAdapter is initialized in glanceclient from here: https://github.com/openstack/python-glanceclient/blob/master/glanceclient/common/http.py#L142

tags: removed: module-ostf
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Decreased to Medium

Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change restored on openstack/keystone (openstack-ci/fuel-7.0/2015.1.0)

Change restored by Boris Bobrov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8189
Reason: still needed

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to openstack/keystone (openstack-ci/fuel-7.0/2015.1.0)

Reviewed: https://review.fuel-infra.org/8189
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 1272e5702ecf7465144d86799d90d510718d5bbc
Author: Alexander Makarov <email address hidden>
Date: Wed Sep 16 14:03:41 2015

Try all available servers on memcache get()

In case of the cache miss try all other servers in the bucket

Change-Id: I59c572b1c572ef5b81670837ec11daa3d1137e14
Related-Bug: 1461036

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/keystone (master)

Related fix proposed to branch: master
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/11662

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/keystone (openstack-ci/fuel-8.0/liberty)

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/12078

Revision history for this message
Alexey Galkin (agalkin) wrote :
Changed in fuel:
status: In Progress → Invalid
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/keystone (openstack-ci/fuel-8.0/liberty)

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/12078
Reason: not needed in 8.0

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/keystone (master)

Change abandoned by Alexander Makarov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/11662
Reason: we don't merge to master

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.