podman inspect : Error: unexpected end of JSON input | openstack wallaby | tripleo

[swogat pradhan]

I was updating the ssl template and then ran openstack overcloud
deploy and am getting error messages as below in most of the nodes:

2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab |
  FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 |
item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo |
error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":
"Failed to run ['/bin/podman', 'inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json'] ['inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json']: time=\"2024-05-07T11:04:10Z\" level=warning
msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected
end of JSON input\n", "prefetch_image":
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"}
2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab |
 TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 |
0:24:39.725509 | 1.78s
2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab |
 TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 |
0:24:39.731144 | 1.78s
2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca |
  FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 |
item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo |
error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":
"Failed to run ['/bin/podman', 'inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json'] ['inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json']: time=\"2024-05-07T11:04:10Z\" level=warning
msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected
end of JSON input\n", "prefetch_image":
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"}
2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca |
 TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 |
0:24:39.737672 | 1.14s
2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca |
 TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 |
0:24:39.741977 | 1.14s
2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e |
 TIMING | Run lvmfilter role | overcloud-novacompute-2 |
0:24:39.745173 | 0.11s
2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca |
  FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 |
item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo |
error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":
"Failed to run ['/bin/podman', 'inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json'] ['inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json']: time=\"2024-05-07T11:04:10Z\" level=warning
msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected
end of JSON input\n", "prefetch_image":
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"}
2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca |
 TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 |
0:24:39.750539 | 1.06s
2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca |
 TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 |
0:24:39.754836 | 1.06s

Whereas in controller nodes I am not getting any errors when doing a
podman inspect using the same id.

[root@overcloud-cephstorage-0 ~]# podman inspect
9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c
--format json
Error: unexpected end of JSON input

Where the same command in controller node gives:

[root@overcloud-controller-0 ~]# podman inspect
9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c
--format json
[
    {
        "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c",
        "Digest":
"sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf",
        "RepoTags": [
            "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"
        ],
        "RepoDigests": [
            "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f",
            "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2022-02-09T05:23:45.932319128Z",
        "Config": {
......

I do not understand what is happening and why this issue is coming.