Pod using the bond CNI plugin experiences an error: "multus-cni-network" failed

[Steven Webster]

Brief Description
-----------------
Using the bond cni plugin as a network attachment definition in a pod causes the pod to fail to launch. A describe on the pod will reveal:

Pod describe error is showing “failed to find plugin "bond" in path [/opt/cni/bin /opt/cni/bin]”

Somewhat recently the plugins were moved to /opt/bin/cni/ from /usr/libexec/cni/ , but the bond plugin was missed.

Severity
--------
Major: System/Feature is usable but degraded

Steps to Reproduce
------------------
The following pod/network attachment definition specs can be used to show the problem:

apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: mypool
spec:
  cidr: "10.10.20.0/24"
  ipipMode: "Never"
  natOutgoing: True
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bond0
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "bond0",
    "type": "bond",
    "ifname": "net1",
    "mode": "active-backup",
    "miimon": "100",
    "failOverMac": 1,
    "links": [
      {
        "name": "eth1000"
      },
      {
        "name": "eth1001"
      }
    ],
    "ipam": {
      "type": "calico-ipam",
      "assign_ipv4": "true",
      "ipv4_pools": ["mypool"]
    },
    "kubernetes": {
      "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
    },
    "datastore_type": "kubernetes"
  }'
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bonding
  namespace: default
  labels:
    tier: node
spec:
  selector:
    matchLabels:
      tier: node
  template:
    metadata:
      labels:
        tier: node
        app: bonding
      annotations:
        cni.projectcalico.org/ipv4pools: '["default-ipv4-ippool"]'
        k8s.v1.cni.cncf.io/networks: '[
                { "name": "bond0" }
        ]'
    spec:
      containers:
      - name: bonding1
        image: centos/tools
        imagePullPolicy: IfNotPresent
        command: [ "/bin/bash", "-c", "--" ]
        args: [ "while true; do sleep 300000; done;" ]
        securityContext:
          capabilities:
            add:
              - NET_ADMIN

Expected Behavior
------------------
One should be able to use the bond plugin in a network attachment definition

Actual Behavior
----------------
The bond plugin cannot be found and the pod will fail to launch.

Reproducibility
---------------
100%

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
The problem would have been seen since commit:

https://opendev.org/starlingx/integ/commit/54f2f7d6c667e0d26211e713d0b1fd44a527cdaa

Last Pass
---------
Anytime since the bond plugin was introduced up to the following commit:

https://opendev.org/starlingx/integ/commit/54f2f7d6c667e0d26211e713d0b1fd44a527cdaa

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Feature Testing

Workaround
----------
Manually move /usr/libexec/cni/bond to /opt/cni/bin/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/843965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/843965
Committed: https://opendev.org/starlingx/integ/commit/478a4e59adacf2ea53a5f66a5119b6e775a67e11
Submitter: "Zuul (22348)"
Branch: master

commit 478a4e59adacf2ea53a5f66a5119b6e775a67e11
Author: Steven Webster <email address hidden>
Date: Fri May 27 13:55:33 2022 -0400

    Fix bond CNI install path

    Recent commit 54f2f7d6c667e0d26211e713d0b1fd44a527cdaa made
    changes to the install path of the containernetworking-plugins
    from /usr/libexec/cni/ to /opt/bin/cni/ as part of making
    /usr readonly to support OSTree.

    Since the bond-cni plugin is not distributed with the other
    containernetworking-plugins, the same change needs to be
    made in the bond-cni package.

    Closes-Bug: 1976111

    Testing:

    Ensure /opt/cni/bin/bond exists on both Debian and CentOS.

    Signed-off-by: Steven Webster <email address hidden>
    Change-Id: I48b47100d14c77818daf42cb24b7146ae6672e35

[Steven Webster (swebster-wr)]

Re-opening as an additional fix must be committed for Debian only as the bond plugin is installed as a directory rather than a binary.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/845872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/845872
Committed: https://opendev.org/starlingx/integ/commit/0c0aa6ff67db885b0a42957126ece93be77f6a36
Submitter: "Zuul (22348)"
Branch: master

commit 0c0aa6ff67db885b0a42957126ece93be77f6a36
Author: Steven Webster <email address hidden>
Date: Tue Jun 14 16:07:58 2022 -0400

    debian: fix issue with bond-cni binary install

    This commit fixes an issue with the debian install of the
    bond CNI plugin.

    It was noticed that the 'bond' source directory was being
    installed, rather than the built 'bond' binary.

    Since the build output is a single binary, it will be
    found in debian/bond-cni/, rather than the standard
    debian/tmp/

    dh_install is instructed where to find the build output
    installed by dh_auto_install.

    Testing:

    - Debian build and install
    - Confirmed the bond binary is at /var/opt/cni/bin/
    - Functional testing to ensure a pod using the
      bond plugin could be launched

    Depends-On: https://review.opendev.org/c/starlingx/integ/+/844865
    Closes-Bug: 1976111

    Signed-off-by: Steven Webster <email address hidden>
    Change-Id: I2a504b8bfc210dc09487a496959da235dec82525