Programming error choosing an endpoint on OVHcloud SGP1

[Raphaël Droz]

Using Ansible to provision SGP1 on OVH public cloud using KeyStone v3:

Traceback (most recent call last):
  File "<stdin>", line 102, in <module>
  File "<stdin>", line 94, in _ansiballz_main
  File "<stdin>", line 40, in invoke_module
  File "/usr/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_os_volume_payload_64uiuf5r/ansible_os_volume_payload.zip/ansible/modules/cloud/openstack/os_volume.py", line 256, in <module>
  File "/tmp/ansible_os_volume_payload_64uiuf5r/ansible_os_volume_payload.zip/ansible/modules/cloud/openstack/os_volume.py", line 248, in main
  File "/tmp/ansible_os_volume_payload_64uiuf5r/ansible_os_volume_payload.zip/ansible/modules/cloud/openstack/os_volume.py", line 155, in _present_volume
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py", line 382, in volume_exists
    return self.get_volume(name_or_id) is not None
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py", line 136, in get_volume
    return _utils._get_entity(self, 'volume', name_or_id, filters)
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_utils.py", line 205, in _get_entity
    entities = search(name_or_id, filters, **kwargs)
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py", line 790, in search_volumes
    volumes = self.list_volumes()
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py", line 78, in list_volumes
    data = self._volume_client.get('/volumes/detail')
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 386, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/proxy.py", line 464, in request
    response = super(_ShadeAdapter, self).request(url, method, **kwargs)
  File "/home/yug/.local/lib/python3.7/site-packages/openstack/proxy.py", line 151, in request
    **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 769, in request
    **endpoint_filter)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1200, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data
    service_name=service_name)
  File "/usr/lib/python3/dist-packages/keystoneauth1/access/service_catalog.py", line 435, in endpoint_data_for
    endpoint_id=endpoint_id)
  File "/usr/lib/python3/dist-packages/keystoneauth1/access/service_catalog.py", line 340, in get_endpoint_data_list
    endpoint_id=endpoint_id)
  File "/usr/lib/python3/dist-packages/keystoneauth1/access/service_catalog.py", line 222, in get_endpoints_data
    return self._endpoints_by_type(service_type, ret)
  File "/usr/lib/python3/dist-packages/keystoneauth1/access/service_catalog.py", line 285, in _endpoints_by_type
    raise ValueError("Programming error choosing an endpoint.")
ValueError: Programming error choosing an endpoint.

https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/access/service_catalog.py#L281

Within _endpoints_by_type, I printed these variables:
- endpoints = {'volumev3': [], 'volumev2': [], 'volume': []}
- alias = block-store
- requested = block-storage

Let me know if I can provide any other information of use.

[Arnaud Morin (arnaud-morin)]

Hello,

Can you share some ansible playbook that you are using to raise the issue?

I cant reproduce using openstack CLI

[Raphaël Droz (raphael-droz)]

Ansible 2.9.4.
Only reproducible with SGP1 (not on 'DE1','BHS3','GRA3','UK1' nor 'SBG3')
- keystoneauth1 3.17.1
- python-keystoneclient 3.21.0.

 To reproduce, a task running `os_volume:` is enough. Mine:

```yaml
- name: 5GB data volume
  os_volume:
    display_name: foo
    display_description: "bar"
    interface: internal
    region_name: SGP1
    size: 5
    verify: True
```

[Monty Taylor (mordred)]

I don't have SGP1 enabled in any of my OVH accounts. Could you run the following script and provide the output (it's going to be verbose - also, it will show HTTP traffic, but the auth tokens are replaced with shas of the tokens so it should be fine to share)

import openstack

openstack.enable_logging(http_debug=True)
c = openstack.connect()
print(c.list_volumes())

[Arnaud Morin (arnaud-morin)]

Thanks Monty for your help.
I checked the commands you gave on my side, and I have no issue.
Note also that the openstack volume list is also working correctly.
By the way, let me know if you need SGP1 on your tenant for debugging, that's something I can do :p

The thing is that the ansible playbook using the task given by Raphael is not working.
On my side I am using:

keystoneauth1==3.18.0
python-keystoneclient==3.22.0
ansible==2.9.4

I will continue digging

[Arnaud Morin (arnaud-morin)]

So, that's because we dont provide the internal endpoint that you are trying to use.
Use this task and it works:
```yaml
- name: 5GB data volume
  os_volume:
    display_name: foo
    display_description: "bar"
    #interface: internal
    region_name: SGP1
    size: 5
    verify: True
```

I think you should not use the internal endpoint, but the public (which is available).

Still there is a question that I want to solve: why dont we have any internal endpoint there?
I will check

[Raphaël Droz (raphael-droz)]

I can confirm that removing "interface: internal" makes the task work. (Still, I think it's worth checking why this only affected SGP1)

Although it's an Ansible question, I don't remember why all my (mostly copy/pasted) os_* Ansible tasks contained this `interface: internal` in the first place : I don't understand the meaning of this option.

[Arnaud Morin (arnaud-morin)]

Hello,

About the missing internal endpoint, I check with the team and it seems to
be a mistake on our side.
We plan the add the internal and admin back into the catalog for SGP1 (and
also SYD1).

About why you have that in your ansible playbooks, that's maybe because you
copy pasted some code from openstack ansible (osa), which, I think, are
using internal endpoint when configuring openstack itself.

Le jeu. 6 févr. 2020 à 17:35, Raphaël Droz <email address hidden> a
écrit :

> I can confirm that removing "interface: internal" makes the task work.
> (Still, I think it's worth checking why this only affected SGP1)
>
> Although it's an Ansible question, I don't remember why all my (mostly
> copy/pasted) os_* Ansible tasks contained this `interface: internal` in
> the first place : I don't understand the meaning of this option.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1861695
>
> Title:
> Programming error choosing an endpoint.
>
> Status in keystoneauth:
> New
>
> Bug description:
> Using Ansible to provision SGP1 on OVH public cloud using KeyStone v3:
>
> Traceback (most recent call last):
> File "<stdin>", line 102, in <module>
> File "<stdin>", line 94, in _ansiballz_main
> File "<stdin>", line 40, in invoke_module
> File "/usr/lib/python3.7/runpy.py", line 205, in run_module
> return _run_module_code(code, init_globals, run_name, mod_spec)
> File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name)
> File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
> exec(code, run_globals)
> File
> "/tmp/ansible_os_volume_payload_64uiuf5r/ansible_os_volume_payload.zip/ansible/modules/cloud/openstack/os_volume.py",
> line 256, in <module>
> File
> "/tmp/ansible_os_volume_payload_64uiuf5r/ansible_os_volume_payload.zip/ansible/modules/cloud/openstack/os_volume.py",
> line 248, in main
> File
> "/tmp/ansible_os_volume_payload_64uiuf5r/ansible_os_volume_payload.zip/ansible/modules/cloud/openstack/os_volume.py",
> line 155, in _present_volume
> File
> "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py",
> line 382, in volume_exists
> return self.get_volume(name_or_id) is not None
> File
> "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py",
> line 136, in get_volume
> return _utils._get_entity(self, 'volume', name_or_id, filters)
> File
> "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_utils.py",
> line 205, in _get_entity
> entities = search(name_or_id, filters, **kwargs)
> File
> "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py",
> line 790, in search_volumes
> volumes = self.list_volumes()
> File
> "/home/yug/.local/lib/python3.7/site-packages/openstack/cloud/_block_storage.py",
> line 78, in list_volumes
> data = self._volume_client.get('/volumes/detail')
> File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line
> 386, in get
> return self.request(url, 'GET', **kwargs)
> ...

[Monty Taylor (mordred)]

Yay! I'm glad that got sorted. I don't think I need SGP1 added anywhere - the internal endpoint thing makes total sense ... but I'll be sure to ping you for access if something else comes up.