kolla-ansible CI: tacker-conductor dies always (Train only)

[Radosław Piliszek]

Since 2019-09-22 tacker-conductor always fails NFV jobs:
https://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-centos-source-scenario-nfv&branch=master

Note: there is no ubuntu variant of this job.

[Radosław Piliszek (yoctozepto)]

Two errors in logs:

2019-09-24 02:44:32.275 7 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images: OSError: [Errno 13] Permission denied: '/var/lib/glance'

2019-09-24 02:44:32.277 7 ERROR tacker.conductor.conductor_server [-] Config option 'vnf_package_csar_path' is not configured correctly. VNF package CSAR path directory /var/lib/tacker/vnfpackages/ doesn't exist

[Radosław Piliszek (yoctozepto)]

These upstream changes seem related (merged during that time and named with 'CSAR'): https://review.opendev.org/#/q/topic:bp/tosca-csar-mgmt-driver+status:merged+project:openstack/tacker

No relevant changes in kolla-ansible.

[Radosław Piliszek (yoctozepto)]

Not marking as critical since stack tests pass otherwise: https://zuul.opendev.org/t/openstack/build/c2cb076178ba405da3587412f4b84223/log/primary/logs/ansible/test-scenario-nfv

[Radosław Piliszek (yoctozepto)]

Example used config for completeness: https://zuul.opendev.org/t/openstack/build/5ba5a5b3611e4e3190a2660557fd0270/log/primary/logs/kolla_configs/tacker-conductor/tacker.conf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/684275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684275
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=b96ade3cf01009d822f85744efee523127f2674c
Submitter: Zuul
Branch: master

commit b96ade3cf01009d822f85744efee523127f2674c
Author: Eduardo Gonzalez <email address hidden>
Date: Tue Sep 24 12:57:58 2019 +0200

    [train] Add required Tacker Conductor config and docs

    Tacker requires config for storing CSAR vnf packages.
    This patch adds it as well as relevant docs.
    Only one Tacker Conductor is deployed by default due to
    lack of a shared filesystem.

    Change-Id: Iad391f35105e79fa9319502256528990915df9b7
    Co-authored-by: Radosław Piliszek <email address hidden>
    Closes-Bug: #1845142

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[Radosław Piliszek (yoctozepto)]

The fix was incomplete.

https://bugs.launchpad.net/kolla-ansible/+bug/1853715

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/695804

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695804
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c5f9ea959092019aa44eb96224fd365b8025bba6
Submitter: Zuul
Branch: master

commit c5f9ea959092019aa44eb96224fd365b8025bba6
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 23 21:46:11 2019 +0100

    [Train] Fix Tacker config for the local Glance store

    It turned out the previous fix ([1]) was incomplete.
    Additionally, it seems we have to limit Tacker server
    to one instance co-located with conductor.

    [1] https://review.opendev.org/684275
    commit b96ade3cf01009d822f85744efee523127f2674c

    Change-Id: I9ce27d5f68f32ef59e245960e23336ae5c5db905
    Closes-bug: #1853715
    Related-bug: #1845142

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/697048

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697048
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c7458b006fe1dc0ccba9460c7090a50a6ac4d3d0
Submitter: Zuul
Branch: stable/train

commit c7458b006fe1dc0ccba9460c7090a50a6ac4d3d0
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 23 21:46:11 2019 +0100

    [Train] Fix Tacker config for the local Glance store

    It turned out the previous fix ([1]) was incomplete.
    Additionally, it seems we have to limit Tacker server
    to one instance co-located with conductor.

    [1] https://review.opendev.org/684275
    commit b96ade3cf01009d822f85744efee523127f2674c

    Change-Id: I9ce27d5f68f32ef59e245960e23336ae5c5db905
    Closes-bug: #1853715
    Related-bug: #1845142
    (cherry picked from commit c5f9ea959092019aa44eb96224fd365b8025bba6)