on k8s master kube-controller-manager not getting cloud-provider / cloud-config cmd line args to

[Jeff Meredith]

i used devstack (latest) to create a k8s cluster via magnum/heat (Fedora Atomic 27):

openstack coe cluster template create k8s-cluster-template \
                           --image fedora-atomic-latest \
                           --keypair testkey \
                           --external-network public \
                           --dns-nameserver 8.8.8.8 \
                           --flavor m1.small \
                           --docker-volume-size 5 \
                           --network-driver flannel \
                           --coe kubernetes \
                           --volume-driver cinder

openstack coe cluster create k8s-cluster \
                      --cluster-template k8s-cluster-template \
                      --master-count 1 \
                      --node-count 1

when i try to deploy MariaDB, the PVC shows an error indicating that the kube_openstack_config file was never read. the claims are stuck in the pending state and when i describe one i see:

  Warning ProvisioningFailed 55s (x26 over 6m) persistentvolume-controller Failed to provision volume with StorageClass "standard": OpenStack cloud provider was not initialized properly : stat /etc/kubernetes/cloud-config: no such file or directory

on the master node i can see the kube-controller-manager process does not receive the --cloud-config parameter (nor the --cloud-provider):

kube 3111 1.8 4.2 141340 86392 ? Ssl Sep19 1:18 /usr/local/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080 --leader-elect=true --service-account-private-key-file=/etc/kubernetes/certs/service_account_private.key --root-ca-file=/etc/kubernetes/certs/ca.crt

the file /etc/kubernetes/controller-manager *does* have them however:

KUBE_CONTROLLER_MANAGER_ARGS="--leader-elect=true --service-account-private-key-file=/etc/kubernetes/certs/service_account_private.key --root-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/kube_openstack_config --cloud-provider=openstack"

it seems as though these args are successfully written out by magnum/cloud-init, but are not picked up when the kube-controller-manager process is launched.

the analogous situation is occurring for the kube-apiserver process.

p.s. heat-coe-templates is probably not the right project for this issue, but it was the closest choice in the pull-down menu.

[Yang Zhang (bjzyang)]

I'm using Magnum rocky release (installed from source code https://github.com/openstack/magnum/tree/stable/rocky) and Fedora Atomic 27, my k8s cluster has exactly the same problem. Please help suggest a solution to this problem other than manually creating a "/etc/kubernetes/cloud-config" file or adding the "--cloud-config=/etc/kubernetes/kube_openstack_config --cloud-provider=openstack" parameters to processes of kube-controller-manager and kube-apiserver. Or did I miss any configuration for Magnum or my k8s cluster?
Thanks!

Yang

[Yang Zhang (bjzyang)]

Btw, the version of Magnum I'm using is 7.0.3.dev1 (from source code https://github.com/openstack/magnum/tree/stable/rocky).

[Yang Zhang (bjzyang)]

More update:
I met this issue in the testing of following scenarios:
* Magnum Rocky 7.0.3.dev1 + Fedora-Atomic-27-20180419.0
* Magnum Rocky 7.0.2 + Fedora-Atomic-27-20180419.0
* Magnum Rocky 7.0.2 + Fedora-Atomic-28-20180806.0

[Christian Zunker (christian-zunker)]

We are running this version of magnum:
# /openstack/venvs/magnum-18.1.0/bin/pip show magnum
Name: magnum
Version: 7.0.2.dev21

OpenStack release is Rocky.

I cannot confirm the missing parameters:
# cat /etc/kubernetes/controller-manager
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--leader-elect=true --service-account-private-key-file=/etc/kubernetes/certs/service_account_private.key --root-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/kube_openstack_config --cloud-provider=openstack"

And it is also picked up by the process:
# ps -ef | grep kube-controller-manage[r]
root 2794 1 0 07:47 ? 00:00:00 /usr/bin/runc --systemd-cgroup run kube-controller-manager
kube 2886 2794 2 07:47 ? 00:07:24 /usr/local/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080 --leader-elect=true --service-account-private-key-file=/etc/kubernetes/certs/service_account_private.key --root-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/kube_openstack_config --cloud-provider=openstack

We are also using Fedora Atomic 27. But we are not using the k8s version default comming with Rocky. Instead we are running with kube_tag=v1.11.5-1