server create fails with "Can not find requested image" when CONF.glance.api_servers is not set and [service_user] is set

Bug #1747511 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Matt Riedemann

Bug Description

http://logs.openstack.org/10/538510/5/check/legacy-tempest-dsvm-neutron-nova-next-full/a03511c/job-output.txt.gz#_2018-02-02_12_25_52_609521

2018-02-02 12:25:52.607290 | primary | Captured traceback:
2018-02-02 12:25:52.607333 | primary | ~~~~~~~~~~~~~~~~~~~
2018-02-02 12:25:52.607396 | primary | Traceback (most recent call last):
2018-02-02 12:25:52.607554 | primary | File "tempest/api/compute/admin/test_auto_allocate_network.py", line 152, in test_server_create_no_allocate
2018-02-02 12:25:52.607674 | primary | self.os_primary, networks='none', wait_until='ACTIVE')
2018-02-02 12:25:52.607798 | primary | File "tempest/common/compute.py", line 191, in create_test_server
2018-02-02 12:25:52.608202 | primary | **kwargs)
2018-02-02 12:25:52.608351 | primary | File "tempest/lib/services/compute/servers_client.py", line 89, in create_server
2018-02-02 12:25:52.608465 | primary | resp, body = self.post('servers', post_body)
2018-02-02 12:25:52.608560 | primary | File "tempest/lib/common/rest_client.py", line 279, in post
2018-02-02 12:25:52.608671 | primary | return self.request('POST', url, extra_headers, headers, body, chunked)
2018-02-02 12:25:52.608792 | primary | File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
2018-02-02 12:25:52.608892 | primary | method, url, extra_headers, headers, body, chunked)
2018-02-02 12:25:52.609032 | primary | File "tempest/lib/common/rest_client.py", line 668, in request
2018-02-02 12:25:52.609111 | primary | self._error_checker(resp, resp_body)
2018-02-02 12:25:52.609225 | primary | File "tempest/lib/common/rest_client.py", line 779, in _error_checker
2018-02-02 12:25:52.609328 | primary | raise exceptions.BadRequest(resp_body, resp=resp)
2018-02-02 12:25:52.609420 | primary | tempest.lib.exceptions.BadRequest: Bad request
2018-02-02 12:25:52.609521 | primary | Details: {u'code': 400, u'message': u'Can not find requested image'}
2018-02-02 12:25:52.609546 | primary |
2018-02-02 12:25:52.609579 | primary |
2018-02-02 12:25:52.609626 | primary | Captured pythonlogging:
2018-02-02 12:25:52.609674 | primary | ~~~~~~~~~~~~~~~~~~~~~~~
2018-02-02 12:25:52.609952 | primary | 2018-02-02 12:05:26,211 2001 INFO [tempest.lib.common.rest_client] Request (AutoAllocateNetworkTest:test_server_create_no_allocate): 400 POST http://198.72.124.140/compute/v2.1/servers 0.381s
2018-02-02 12:25:52.610298 | primary | 2018-02-02 12:05:26,212 2001 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json', 'X-OpenStack-Nova-API-Version': '2.37'}
2018-02-02 12:25:52.610880 | primary | Body: {"server": {"imageRef": "b14816be-0047-4749-964f-be5f5eaef245", "networks": "none", "flavorRef": "42", "name": "tempest-tempest.common.compute-instance-166758257"}}
2018-02-02 12:25:52.611673 | primary | Response - Headers: {u'connection': 'close', u'content-type': 'application/json; charset=UTF-8', u'server': 'Apache/2.4.18 (Ubuntu)', u'x-compute-request-id': 'req-98f8b35e-a065-4c10-b37f-073d7bde3822', u'openstack-api-version': 'compute 2.37', 'status': '400', 'content-location': 'http://198.72.124.140/compute/v2.1/servers', u'content-length': '72', u'x-openstack-nova-api-version': '2.37', u'x-openstack-request-id': 'req-98f8b35e-a065-4c10-b37f-073d7bde3822', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'date': 'Fri, 02 Feb 2018 12:05:25 GMT'}
2018-02-02 12:25:52.611797 | primary | Body: {"badRequest": {"message": "Can not find requested image", "code": 400}}

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Can%20not%20find%20requested%20image%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=10d

So it's using b14816be-0047-4749-964f-be5f5eaef245 which is created here:

http://logs.openstack.org/10/538510/5/check/legacy-tempest-dsvm-neutron-nova-next-full/a03511c/logs/devstacklog.txt.gz#_2018-02-02_12_01_36_458

2018-02-02 12:01:36.458 | + functions:upload_image:380 : openstack --os-cloud=devstack-admin --os-region-name=RegionOne image create cirros-0.3.5-x86_64-disk --public --container-format=bare --disk-format qcow2
2018-02-02 12:01:40.203 | +------------------+------------------------------------------------------+
2018-02-02 12:01:40.203 | | Field | Value |
2018-02-02 12:01:40.203 | +------------------+------------------------------------------------------+
2018-02-02 12:01:40.204 | | checksum | f8ab98ff5e73ebab884d80c9dc9c7290 |
2018-02-02 12:01:40.204 | | container_format | bare |
2018-02-02 12:01:40.204 | | created_at | 2018-02-02T12:01:39Z |
2018-02-02 12:01:40.204 | | disk_format | qcow2 |
2018-02-02 12:01:40.204 | | file | /v2/images/b14816be-0047-4749-964f-be5f5eaef245/file |
2018-02-02 12:01:40.204 | | id | b14816be-0047-4749-964f-be5f5eaef245 |
2018-02-02 12:01:40.204 | | min_disk | 0 |
2018-02-02 12:01:40.204 | | min_ram | 0 |
2018-02-02 12:01:40.204 | | name | cirros-0.3.5-x86_64-disk |
2018-02-02 12:01:40.204 | | owner | b5e61d54c73a4ec388e770d352f75d13 |
2018-02-02 12:01:40.204 | | protected | False |
2018-02-02 12:01:40.204 | | schema | /v2/schemas/image |
2018-02-02 12:01:40.204 | | size | 13267968 |
2018-02-02 12:01:40.204 | | status | active |
2018-02-02 12:01:40.204 | | tags | |
2018-02-02 12:01:40.204 | | updated_at | 2018-02-02T12:01:40Z |
2018-02-02 12:01:40.204 | | virtual_size | None |
2018-02-02 12:01:40.204 | | visibility | public |
2018-02-02 12:01:40.204 | +------------------+------------------------------------------------------+

And set in tempest.conf:

http://logs.openstack.org/10/538510/5/check/legacy-tempest-dsvm-neutron-nova-next-full/a03511c/logs/tempest_conf.txt.gz

[compute]
max_microversion = latest
flavor_ref_alt = 84
flavor_ref = 42
image_ref_alt = b14816be-0047-4749-964f-be5f5eaef245
image_ref = b14816be-0047-4749-964f-be5f5eaef245
build_timeout = 196

I'm not sure what changed around 1/31 but this is failing on multiple changes, only on master, and only for the nova-next job.

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

The only thing unique about the nova-next job is it sets NOVA_USE_SERVICE_TOKEN=True in devstack, so maybe there is something wrong with how we're constructing the keystoneauth1 adapter for the image service?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/541008

Revision history for this message
Matt Riedemann (mriedem) wrote : Re: legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find requested image" since 1/31

http://logs.openstack.org/08/541008/2/check/legacy-tempest-dsvm-neutron-nova-next-full/f24c9d2/logs/screen-n-api.txt.gz#_Feb_05_21_46_51_455934

Feb 05 21:46:51.455934 ubuntu-xenial-ovh-bhs1-0002377979 <email address hidden>[19654]: INFO nova.image.glance [None req-15a828fe-6d9b-4f80-b599-a4a234f9806c tempest-AggregatesAdminTestJSON-1418914203 tempest-AggregatesAdminTestJSON-1418914203] Using glance API servers: [u'http://158.69.92.100/image/v2/']

http://logs.openstack.org/08/541008/2/check/legacy-tempest-dsvm-neutron-nova-next-full/f24c9d2/logs/screen-g-api.txt.gz#_Feb_05_21_46_51_651885

Feb 05 21:46:51.651885 ubuntu-xenial-ovh-bhs1-0002377979 <email address hidden>[18838]: [pid: 18841|app: 0|req: 5/11] 127.0.0.1 () {40 vars in 1011 bytes} [Mon Feb 5 21:46:51 2018] GET /v2/v2/images/d5659eaf-1c56-4b35-970b-229a91b84bbd => generated 154 bytes in 90 msecs (HTTP/1.1 404) 4 headers in 172 bytes (1 switches on core 0)

Which I guess is a glanceclient bug:

https://bugs.launchpad.net/python-glanceclient/+bug/1707995

Revision history for this message
Matt Riedemann (mriedem) wrote :

The problem definitely has something to do with setting up service user credentials, because in this patch https://review.openstack.org/#/c/541008/ I removed the code that handles the service user config settings and now the nova-next job passes.

summary: - legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find
- requested image" since 1/31
+ server create fails with "Can not find requested image" when
+ CONF.glance.api_servers is not set and [service_user] is set
Changed in nova:
status: New → Confirmed
Revision history for this message
Matt Riedemann (mriedem) wrote :

We will likely need to put out a known issue release note for queens for this regression.

Changed in nova:
importance: Undecided → High
tags: added: queens-rc-potential
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/541008
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=62ef6cfcf01d84813f71d1e8252b86c170ee39f0
Submitter: Zuul
Branch: master

commit 62ef6cfcf01d84813f71d1e8252b86c170ee39f0
Author: Matt Riedemann <email address hidden>
Date: Mon Feb 5 16:07:28 2018 -0500

    Workaround glanceclient bug when CONF.glance.api_servers not set

    In certain configurations, like when setting [service_user]
    config, and not setting [glance]/api_servers, the KSA adapter
    get endpoint code (new in Queens) will return a versioned URL
    which glanceclient doesn't handle (due to bug 1707995) so we
    need to workaround that by parsing the URL to strip the version
    from the endpoint URL we got from KSA.

    This is validated in the nova-next CI job which configures a
    service user token for glance.

    Change-Id: I363182e916480c734cc37f279e8e89c8f3ec653c
    Closes-Bug: #1747511
    Related-Bug: #1707995

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0rc1

This issue was fixed in the openstack/nova 17.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/549893

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/549893
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=21be5060e15838da96f6215d978c5dd2a5e6c4f7
Submitter: Zuul
Branch: master

commit 21be5060e15838da96f6215d978c5dd2a5e6c4f7
Author: Matt Riedemann <email address hidden>
Date: Mon Mar 5 18:47:48 2018 -0500

    Make the nova-next job voting and gating

    Bug 1747511 was fixed with change
    I363182e916480c734cc37f279e8e89c8f3ec653c in Queens
    so we can make the nova-next job voting now.

    We want this job voting and gating since it tests
    some things not found in other job configurations,
    like TLS proxy, service user tokens, and post-test hooks
    for nova-manage commands like archive_deleted_rows.

    Change-Id: I4cfc3b100f8829deb97e491f40a3020b3c8eb026
    Related-Bug: #1747511

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/620888

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/620888
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e1f08660d4c77ecbbff2066234ba152961e94502
Submitter: Zuul
Branch: stable/queens

commit e1f08660d4c77ecbbff2066234ba152961e94502
Author: Matt Riedemann <email address hidden>
Date: Mon Mar 5 18:47:48 2018 -0500

    Make the nova-next job voting and gating

    Bug 1747511 was fixed with change
    I363182e916480c734cc37f279e8e89c8f3ec653c in Queens
    so we can make the nova-next job voting now.

    We want this job voting and gating since it tests
    some things not found in other job configurations,
    like TLS proxy, service user tokens, and post-test hooks
    for nova-manage commands like archive_deleted_rows.

    Conflicts:
          .zuul.yaml
    Conflicts are due to https://review.openstack.org/#/c/604134/

    Change-Id: I4cfc3b100f8829deb97e491f40a3020b3c8eb026
    Related-Bug: #1747511
    (cherry picked from commit 21be5060e15838da96f6215d978c5dd2a5e6c4f7)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.