Fail to attach a volume using kubernetes cloud provider

[Markus Sommer]

When attempting to attach a cinder volume to a node, the node-controller-manager needs to first negotiate the API-version with cinder. As of Kubernetes v1.7, the controller appears unable to do so. Attempting to deploy a pod will fail with the error

FailedMount

AttachVolume.Attach failed for volume "myvolume" : Expected HTTP response code [200 204] when accessing [GET https://openstack.cern.ch:8776/], but got 300 instead

{"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "http://openstack.cern.ch:8776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "http://openstack.cern.ch:8776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "http://openstack.cern.ch:8776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.27", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]}

As a workaround, one can force an API-version by adding the following to /etc/sysconfig/kube_openstack_config

[BlockStorage]
bs-version=v2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/483970

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483970
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=94d93177dbc7c6b3653cda166e0cb96f68f91f5c
Submitter: Jenkins
Branch: master

commit 94d93177dbc7c6b3653cda166e0cb96f68f91f5c
Author: Markus Sommer <email address hidden>
Date: Fri Jul 14 17:23:25 2017 +0200

    Add Cinder-API-ver to k8s-cloud-provider config

    Explicit API-version is required when communicating with cinder, since
    kubernetes expects a return 200 but receives 300 wen version is not
    specified.

    Change-Id: I1202d85f8029a078c099eee105b421c9660393fc
    Closes-Bug: #1704410

[Mohammed Naser (mnaser)]

This has broken Fedora Atomic 26 support as it ships with Kubernetes 1.6 which refuses to start because this option does not exist in Kubernetes 1.6.

[Mohammed Naser (mnaser)]

After researching and with help from dims, one of the Kubernetes OpenStack provider maintainers, he's brought up this issue:

https://github.com/kubernetes/kubernetes/pull/47602

This will be fixed in 1.7.3 once it is released. We are breaking almost every release (and I'm not even sure how you ended up with 1.7.x considering Fedora Atomic goes up to 1.6.x?)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/489657

[Spyros Trigazis (strigazi)]

I think we don't need [1] anymore. We just need to revert [2], correct?

[1] https://review.openstack.org/#/c/489657/
[2] https://review.openstack.org/#/c/483970/

[Mohammed Naser (mnaser)]

As discussed with Spyros on IRC, we discussed the following:

- We control the Kubernetes releases
- We will make sure we ship Kubernetes >= 1.7.3 if we ever do
- Proposing a revert of the commit to fix this bug
- Add a release note warning to use 1.7.3 or newer if running 1.7.X

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mohammed Naser (<email address hidden>) on branch: master
Review: https://review.openstack.org/489657

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/magnum 5.0.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mohammed Naser (<email address hidden>) on branch: master
Review: https://review.openstack.org/489657