Ubuntu 22.04.1 LTS libpod (package podman 3.4.4+ds1-1ubuntu1.22.04.1): broken network functionality for CNI plugins

[Artem]

After upgrading to the podman 3.4.4+ds1-1ubuntu1.22.04.1 all networks (other than the default podman network) creates with cniVersion 1.0.0 and it follows to an error described bellow:

root@ubuntu:/home/user# podman network create -d bridge test-net
/etc/cni/net.d/test-net.conflist
root@ubuntu:/home/user# podman network ls
WARN[0000] Error validating CNI config file /etc/cni/net.d/test-net.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]
NETWORK ID NAME VERSION PLUGINS
2f259bab93aa podman 0.4.0 bridge,portmap,firewall,tuning
254ddbe0ec1e test-net 1.0.0 bridge,portmap,firewall,tuning,dnsname

In the previous version (podman=3.4.4+ds1-1ubuntu1) to create networks in the podman were using 0.4.0 CNI version.

---

root@ubuntu:/home/user# podman run -it --network 254ddbe0ec1e alpine:latest /bin/sh
WARN[0000] Error validating CNI config file /etc/cni/net.d/test-net.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 31e352740f53 done
Copying config c1aabb73d2 done
Writing manifest to image destination
Storing signatures
WARN[0002] Error validating CNI config file /etc/cni/net.d/test-net.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]
ERRO[0002] error loading cached network config: network "test-net" not found in CNI cache
WARN[0002] falling back to loading from existing plugins on disk
WARN[0002] Error validating CNI config file /etc/cni/net.d/test-net.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]
ERRO[0002] Error tearing down partially created network namespace for container 3a31651748e7477ebe2dae5465d875537890e2a9099a0e234223a2a6ed211eca: CNI network "test-net" not found
Error: error configuring network namespace for container 3a31651748e7477ebe2dae5465d875537890e2a9099a0e234223a2a6ed211eca: CNI network "test-net" not found

---

cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

---

podman version
WARN[0000] Error validating CNI config file /etc/cni/net.d/test-net.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.18.1
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64

---

dpkg -l podman
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==========================-============-==========================================
ii podman 3.4.4+ds1-1ubuntu1.22.04.1 amd64 engine to run OCI-based containers in Pods

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Henrique Luis Schmidt (henrique-schmidt93)]

After manually installing the version containernetworking-plugins_1.1.1+ds1-1_amd64.deb, downloaded from this link http://archive.ubuntu.com/ubuntu/pool/universe/g/golang-github-containernetworking-plugins/, the problem was solved.

[Jeremy (jeremyfritzen)]

Hi!

Thanks.
What are the impacts of installing an archived package?

[joseche (joseche)]

Confirming this bug:

# dpkg -l containernetworking-plugins podman podman-docker
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===========================-==========================-============-===============================================================
ii containernetworking-plugins 0.9.1+ds1-1 amd64 standard networking plugins - binaries
ii podman 3.4.4+ds1-1ubuntu1.22.04.1 amd64 engine to run OCI-based containers in Pods
ii podman-docker 3.4.4+ds1-1ubuntu1.22.04.1 amd64 engine to run OCI-based containers in Pods - wrapper for docker

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

Also, thanks to @henrique-schmidt93, to fix it just do this:

curl -O http://archive.ubuntu.com/ubuntu/pool/universe/g/golang-github-containernetworking-plugins/containernetworking-plugins_1.1.1+ds1-1_amd64.deb

dpkg -i containernetworking-plugins_1.1.1+ds1-1_amd64.deb

[Zastrix Arundell (zastrix)]

Happening as well on a Ubuntu 22.04.2 LTS aarch64 based system. I can't say whether the fix above works as there's no package to run on an ARM based machine.

[Felix Herrmann (felher)]

Can confirm this bug as well as the fix.

root@Ansible-test:/home/fherrmann# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

[Dezim Kitt (dezim-kitt)]

Any news when this issue will be solved? Our current infrastructure running Ubuntu and Podman is affected from this bug.

[Artem (ateleshev)]

@dezim-kitt
Not sure it will be fixed quickly, you may switch to Debian 12 (current stable). You will get the newest kernel with podman 4.3 instead of 3.7

[Artem (ateleshev)]

looks like this bug never be closed, switched to the debian:12

[Zastrix Arundell (zastrix)]

Likewise. Changed my infrastructure to debian:12 as well.

[Joe Madden (joemadden1989)]

Is this going to be fixed anytime soon? Its a pretty fundemetal bug that has been outstanding for quite sometime.

Work around:

Downgrade to the previous version - I don't know what security fixes this patch is missing so at your own risk:
sudo apt install podman=3.4.4+ds1-1ubuntu1

[James V (saracen9)]

I found that using the `Kubic Repo` and upgrading to a newer version of the package fixed this for me.

See: https://podman.io/docs/installation#ubuntu

```
sudo mkdir -p /etc/apt/keyrings
curl -fsSL "https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_$(lsb_release -rs)/Release.key" \
  | gpg --dearmor \
  | sudo tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]\
    https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_$(lsb_release -rs)/ /" \
  | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null
sudo apt-get update -qq
sudo apt-get -qq -y install podman
```

[Rick (rickepnet)]

I agree there's no packages for the arm processor. Did anybody find a solution that works with arm?

curl -O http://archive.ubuntu.com/ubuntu/pool/universe/g/golang-github-containernetworking-plugins/containernetworking-plugins_1.1.1+ds1-1_amd64.deb

[Ben (ltm-linux)]

From what I can tell, any networks I had running with CNI version 0.4.0 were running fine, even after podman updated to 3.4.4. I only had the issue when I created a new interfaces while running podman 3.4.4, at which point the configuration for those new network interfaces were trying to use CNI version 1.0.0 instead of 0.4.0.

To resolve this, instead of downgrading the podman version (or upgrading podman from a different repo), I simply changed the CNI version being used by the network I created. I did this by manually editing the .conflist file in the ~/.config/cni/net.d/ directory for the network I had created, changing the "cniVersion" field from "1.0.0" to "0.4.0".

Once I did this, everything seemed to work for me and I no longer had the error messages appearing.

Since it doesn't involve changing the installed podman version or any of the other installed packages, this should be a workaround that works with ARM processor.

Hope that helps..

[Artur (tambel)]

Thanks for all provided workarounds.
This is really sad that this bug has made it to 2024 and still not fixed.

[Alvin Ng (alvinsj)]

update link:

curl -O http://archive.ubuntu.com/ubuntu/pool/universe/g/golang-github-containernetworking-plugins/containernetworking-plugins_1.1.1+ds1-3build1_amd64.deb

dpkg -i containernetworking-plugins_1.1.1+ds1-3build1_amd64.deb

[Jean Lescure (jeanlescure)]

I'm having the same problem, downloading and installing the containernetworking-plugins provided by @alvinsj fixed it.

I'm guessing podman is not ready for production usage, will have to revert to docker for our project.

[Ralf Hansen (abacab42)]

I can confirm this error and the workarounds.

The workaround to correct cniVersion in ~/.config/cni/net.d/*.conflist is working for me but it isnn't very intuitive for others.
The universe package containernetworking-plugins corrects this too but thats not easy maintable for a production system over time.

This is a showstopper for ubuntu container tools.

[Nico (nicc777)]

Ran into this issue today. Surprised this is still open.

Anyway, I tried the solution from #16 (thanks Alvin) and it worked.

[Dawson Pleasant (djpleasant)]

Ran into this issue on Ubuntu 22.04 created with Windows 11 Hyper-V "quick create".

I created a network using:
`podman network create <my-network>`

Any further podman commands would raise the warning listed in this post.

I used @abacab42's solution from #18 to resolve the issue by changing the "cniVersion" property to "0.4.0" from "1.0.0" in the `~/.config/cni/net.d/<my_network>.conflist` file.

I was then able to successfully spin up my networked container without any warnings.

I'll also echo @abacab42's last statement; this is a bummer of a bug.