[UBUNTU 20.04] nfs storage failed to run on top of kubernetes-core cluster on s390x

[bugproxy]

Steps to reproduce:
LXD setup and juju installation are done following these instructions https://ubuntu.com/kubernetes/docs/install-local

Kubernetes Core bundle is installed as described here https://jaas.ai/kubernetes-core with
juju deploy cs:bundle/kubernetes-core-1069

Install nfs as it is mentioned at https://ubuntu.com/kubernetes/docs/storage
juju deploy nfs --constraints root-disk=10G
juju add-relation nfs kubernetes-worker

Expected result:
`kubectl get sc,po` will show running pod and storage class

Actual result:
# kubectl get sc,po
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/default (default) fuseim.pri/ifs Delete Immediate false 2m17s

NAME READY STATUS RESTARTS AGE
pod/nfs-client-provisioner-7698b8dbc9-wj56n 0/1 ContainerCreating 0 2m17s

# kubectl describe pod nfs-client-provisioner-7698b8dbc9-wj56n
Name: nfs-client-provisioner-7698b8dbc9-wj56n
Namespace: default
Priority: 0
Node: juju-62238e-2/10.70.13.80
Start Time: Fri, 31 Jul 2020 11:39:19 +0200
Labels: app=nfs-client-provisioner
                pod-template-hash=7698b8dbc9
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nfs-client-provisioner-7698b8dbc9
Containers:
  nfs-client-provisioner:
    Container ID:
    Image: rocks.canonical.com:443/cdk/external_storage/nfs-client-provisioner:v3.1.0-k8s1.11
    Image ID:
    Port: <none>
    Host Port: <none>
    State: Waiting
      Reason: ContainerCreating
    Ready: False
    Restart Count: 0
    Environment:
      PROVISIONER_NAME: fuseim.pri/ifs
      NFS_SERVER: 10.70.13.33
      NFS_PATH: /srv/data/kubernetes-worker
    Mounts:
      /persistentvolumes from nfs-client-root (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-r2qgb (ro)
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  nfs-client-root:
    Type: NFS (an NFS mount that lasts the lifetime of a pod)
    Server: 10.70.13.33
    Path: /srv/data/kubernetes-worker
    ReadOnly: false
  default-token-r2qgb:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-r2qgb
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled <unknown> default-scheduler Successfully assigned default/nfs-client-provisioner-7698b8dbc9-wj56n to juju-62238e-2
  Warning FailedMount 2m58s kubelet, juju-62238e-2 MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/98db0283-95db-47fc-911e-e4375fd9f6cf/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs 10.70.13.33:/srv/data/kubernetes-worker /var/lib/kubelet/pods/98db0283-95db-47fc-911e-e4375fd9f6cf/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r946191ed443941e5ad6dcc42ede13325.scope
mount.nfs: requested NFS version or transport protocol is not supported
  Warning FailedMount 2m57s kubelet, juju-62238e-2 MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/98db0283-95db-47fc-911e-e4375fd9f6cf/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs 10.70.13.33:/srv/data/kubernetes-worker /var/lib/kubelet/pods/98db0283-95db-47fc-911e-e4375fd9f6cf/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r5291c4d534b54caa9a2a508e128f00af.scope
mount.nfs: requested NFS version or transport protocol is not supported
  Warning FailedMount 2m56s kubelet, juju-62238e-2 MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
....

So nfs pod actually fails to start because of "requested NFS version or transport protocol is not supported"

[Frank Heimes (fheimes)]

I'm not sure if kubernetes-core is supposed to run on s390x - leaving that to the CDK team.
But I did the deployment with: 'juju deploy cs:canonical-kubernetes' - at least of v1.16 (some time ago).

[Tim Van Steenburgh (tvansteenburgh)]

Frank I think this should be filed against https://launchpad.net/charmed-kubernetes-bundles instead of "CDK", which is an unrelated project.

[Frank Heimes (fheimes)]

Thx for the hint, Tim. Updated...

[George Kraft (cynerva)]

kubernetes-core on s390x should be fine.

The "requested NFS version or transport protocol is not supported" suggests that there may be a problem with the NFS server. If you can, please share output of the following commands:

juju status --format yaml
juju debug-log -i nfs --replay
juju run --unit nfs/0 -- journalctl -o cat -u nfs-kernel-server

Even if we get past the mount issue, I expect the nfs-client-provisioner pod will fail to start because rocks.canonical.com:443/cdk/external_storage/nfs-client-provisioner:v3.1.0-k8s1.11 is not a multiarch image.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-10-20 08:21 EDT-------
Test only comment, if mirror works.

[Heinz-Werner Seeck (heinz-werner-seeck)]

Test only comment, if mirror to BZ works.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-11-13 05:21 EDT-------
This LP will be closed. Currently no possibility to create addl. information regarding this issue. If this problem will occur in the future again, a new LP will be created...

------- Comment From <email address hidden> 2020-11-13 05:22 EDT-------
IBM Bugzilla->closed.