podman 1.5 - json-file logging specified but not supported. Choosing k8s-file logging instead

Bug #1844856 reported by Cagri Ersen
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Cédric Jeanneret

Bug Description

Description
===========
Overcloud deployment fails with below error when using overcloud-full.tar image which downloaded from rdo_trunk [https://images.rdoproject.org/master/rdo_trunk/current-tripleo/overcloud-full.tar]

```
ERROR: 28409 -- ['/usr/bin/podman', 'start', '-a', u'container-puppet-nova_libvirt'] run failed after time=\"2019-09-21T00:43:03+03:00\" level=error msg=\"json-file logging specified but not supported. Choosing k8s-file logging instead\
```

I think this is related to podman version 1.5, which doesn't have json-file logging driver. The image includes this version.

The undercloud has podman version 1.4 and if I built the overcloud-full image in undercloud then overcloud nodes have this version as well.

Steps to reproduce
==================
1 - Build container images with `openstack tripleo container image prepare default --local-push-destination --output-env-file ~/templates/containers-prepare-parameter.yaml`

2 - Install overcloud

3 - Check undercloud's podman version:

```
(undercloud) [stack@undercloud ~]$ sudo podman info
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-2.el7.centos.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.3.0, commit: unknown'
```

3 - Check one of the container's LogConfig information by inspecting it.

```
(undercloud) [stack@undercloud ~]$ sudo podman inspect 99dfb9997f099
        "HostConfig": {
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
```

3 - Download overcloud-full image from rdo_trunk and upload it to glance.

```
(undercloud) [stack@undercloud ~]$ cd ~/images

(undercloud) [stack@undercloud ~]$ curl -o ironic-python-agent.tar https://images.rdoproject.org/master/rdo_trunk/current-tripleo/ironic-python-agent.tar

(undercloud) [stack@undercloud ~]$ curl -o overcloud-full.tar https://images.rdoproject.org/master/rdo_trunk/current-tripleo/overcloud-full.tar
for x in overcloud-full.tar ironic-python-agent.tar; do tar -xvf $x ; done
```

4 - Run overcloud deployment.

Expected result
===============
Podman should run the containers on overcloud nodes.

Actual result
=============
Deployment fails with an error about to logging driver.

```
ERROR: 28409 -- ['/usr/bin/podman', 'start', '-a', u'container-puppet-nova_libvirt'] run failed after time=\"2019-09-21T00:43:03+03:00\" level=error msg=\"json-file logging specified but not supported. Choosing k8s-file logging instead\
```

Environment
===========
1. Version: Stein

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.opendev.org/683937

Changed in tripleo:
assignee: nobody → Cédric Jeanneret (cjeanner)
status: New → In Progress
Changed in tripleo:
importance: Undecided → High
milestone: none → train-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.opendev.org/683937
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0976e4eeb3421873ad487d01ce85b0ec94d735ce
Submitter: Zuul
Branch: master

commit 0976e4eeb3421873ad487d01ce85b0ec94d735ce
Author: Cédric Jeanneret <email address hidden>
Date: Mon Sep 23 13:49:31 2019 +0200

    Update log-driver value for podman

    Depending on the podman version, "json-file" is set to noop and makes
    podman crash (true for at least podman 1.4.1), while older versions
    re-add this json-file as an alias to k8s-file (true since 1.4.3).

    Ensuiring we're using k8s-file will prevent issues depending on the
    podman version.

    Relates to https://bugzilla.redhat.com/show_bug.cgi?id=1754416
    Closes-Bug: #1844856

    Change-Id: I70eba8af06741ed81173689a03c4867421917cd6

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/684827

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/stein)

Reviewed: https://review.opendev.org/684827
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0c9066661756222b7902c33ed0532abcb36f3b89
Submitter: Zuul
Branch: stable/stein

commit 0c9066661756222b7902c33ed0532abcb36f3b89
Author: Cédric Jeanneret <email address hidden>
Date: Mon Sep 23 13:49:31 2019 +0200

    Update log-driver value for podman

    Depending on the podman version, "json-file" is set to noop and makes
    podman crash (true for at least podman 1.4.1), while older versions
    re-add this json-file as an alias to k8s-file (true since 1.4.3).

    Ensuiring we're using k8s-file will prevent issues depending on the
    podman version.

    Relates to https://bugzilla.redhat.com/show_bug.cgi?id=1754416
    Closes-Bug: #1844856

    Change-Id: I70eba8af06741ed81173689a03c4867421917cd6
    (cherry picked from commit 0976e4eeb3421873ad487d01ce85b0ec94d735ce)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 11.3.0

This issue was fixed in the openstack/puppet-tripleo 11.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 10.5.2

This issue was fixed in the openstack/puppet-tripleo 10.5.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.