Nova compute api v2.1/servers in train

[BN]

**Environment**:
* OS (e.g. from /etc/os-release): Ubuntu
* Kernel (e.g. `uname -a`): Linux host 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
* Docker version if applicable (e.g. `docker version`): 19.03.2
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): 9.0.0
* Docker image Install type (source/binary): source
* Docker image distribution: train
* Are you using official images from Docker Hub or self built? official
* If self built - Kolla version and environment used to build:
* Share your inventory file, globals.yml and other configuration files if relevant

-------------------------

I have updated kolla-ansible(to 9.0.0) and openstack images(to train) recently. Thus, I was using Rancher node driver to provision openstack instances and use it to deploy k8s cluster. With Stein everything was working smoothly. However, after I updated to Train version, Rancher started getting 400-403 error codes:

```
Error creating machine: Error in driver during machine creation: Expected HTTP response code [200] when accessing [POST http://10.0.225.254:8774/v2.1/os-keypairs], but got 403 instead

or

Error creating machine: Error in driver during machine creation: Expected HTTP response code [200] when accessing [POST http://10.0.225.254:8774/v2.1/servers], but got 400 instead
```

Thus, I am wondering if anything was changed to nova compute api's in Train version and what action can be done in order to fix that issue? I have reported that bug on Rancher github as well: https://github.com/rancher/rancher/issues/24813 cause I am not sure if its fully openstack-version related issue.

Regards

[Radosław Piliszek (yoctozepto)]

Well, API is versioned to keep compatibility constraints. This is not Kolla-specific issue, notifying Nova.

[Ghanshyam Mann (ghanshyammann)]

In Nova API, we did few changes during Train. Main change in API was returning the error for unknown param for query param and for request body which were ignored till stein.

Can you paste the complete log of request and response for those error to confirm if that is valid error ?

[BN (zatoichy)]

Hello,

any update on this :(

[Stephen Finucane (stephenfinucane)]

The main change I can see in stable/train is the inclusion of API microversion 2.75, which made the API stricter and means you will now get a "400 error response for an unknown parameter in the querystring or request body" [1]. This is correct behavior from nova's perspective and it's rancher than needs to be fixed. You can get more information by looking at the body for the 4xx responses and the logs for the nova-api services.

[1] https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id68

[melanie witt (melwitt)]

Looking at the details of the Rancher bug [1] you opened, you mentioned:

"Other details that may be helpful:

I assume, it can be fixed if API's support new version of Openstack:Train.

If I run - openstack:8774/v2.1/servers - unauthorized error / non-existent API.
If I run - openstack:8774/compute/v2.1/servers - it works. [ https://docs.openstack.org/api-ref/compute/ ]"

You're seeing a difference in the behavior based on the service URL for nova [2] -- 'openstack:8774' vs 'openstack:8774/compute'. The service URL comes from the service catalog which resides in the identity service (keystone). Whatever is calling the nova API should be extracting the service URL for nova from the identity token (from keystone) and using that instead of hard-coding a service URL.

It sounds like the application calling the nova API has hard-coded the service URL and when the Train version was deployed, a different/new service URL was populated in the service catalog for the compute service (nova).

[1] https://github.com/rancher/rancher/issues/24813
[2] https://docs.openstack.org/api-ref/compute/#service-urls

[BN (zatoichy)]

Hi Ghanshyam,

Thank you for your response.

Complete log would be the same: https://pastebin.com/mRHh6Th7

docker-machine cannot connect to Nova api. However, when using Stein it was working fine without any issues.

Thank you

[BN (zatoichy)]

https://github.com/rancher/machine/tree/master/drivers/openstack - I found that driver, I would appreciate if you can suggest what can be changed in order it to read nova correctly. Shall I change v2 to v2.1/compute?

Thank you

[Tom Fifield (fifieldt)]

Since this appears to be an issue with Rancher or an application rather than Kolla Ansible, marking as invalid,.