etcd fails to start, ENV missing data-dir, permissions denied to default data-dir

[Charlie Kang]

Been attempting to use branch stable/ocata for `kolla` & `kolla-ansible`
CentOS 7.3

Debugging what might have changed but etcd container but it is failing to start.

Current failure:
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
Running command: 'etcd'
2017-03-01 10:06:04.161480 I | etcdmain: etcd Version: 3.0.15
2017-03-01 10:06:04.161591 I | etcdmain: Git SHA: fc00305
2017-03-01 10:06:04.161605 I | etcdmain: Go Version: go1.6.3
2017-03-01 10:06:04.161624 I | etcdmain: Go OS/Arch: linux/amd64
2017-03-01 10:06:04.161667 I | etcdmain: setting maximum number of CPUs to 56, total number of available CPUs is 56
2017-03-01 10:06:04.161683 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2017-03-01 10:06:04.162519 I | etcdmain: listening for peers on http://localhost:2380
2017-03-01 10:06:04.162803 I | etcdmain: listening for client requests on localhost:2379
2017-03-01 10:06:04.162940 I | etcdmain: stopping listening for client requests on localhost:2379
2017-03-01 10:06:04.163006 I | etcdmain: stopping listening for peers on http://localhost:2380
2017-03-01 10:06:04.163049 C | etcdmain: cannot access data directory: mkdir default.etcd: permission denied

Looking at deployment logs, environment ETCD_DATA_DIR is set along with other variables.

TASK [etcd : Ensuring config directories exist] ********************************
changed: [au11-10-wcp] => (item={'key': u'etcd', 'value': {u'group': u'etcd', u'image': u'10.166.30.60:4000/ocata/centos-source-etcd:2.4rc1', u'enabled': True, u'environment': {u'ETCD_ADVERTISE_CLIENT_URLS': u'http://10.11.117.85:2379', u'ETCD_INITIAL_CLUSTER_STATE': u'new', u'ETCD_INITIAL_ADVERTISE_PEER_URLS': u'http://10.11.117.85:2380', u'ETCD_INITIAL_CLUSTER': u'au11-10-wcp=http://10.11.117.85:2380', u'ETCD_DATA_DIR': u'/var/lib/etcd', u'KOLLA_CONFIG_STRATEGY': u'COPY_ALWAYS', u'ETCD_LISTEN_PEER_URLS': u'http://10.11.117.85:2380', u'ETCD_INITIAL_CLUSTER_TOKEN': u'RYpEl4zUZJfGAvMtVcOA9fnloL5CdRENtf0BWlAv', u'ETCD_NAME': u'au11-10-wcp', u'ETCD_LISTEN_CLIENT_URLS': u'http://10.11.117.85:2379', u'ETCD_OUT_FILE': u'/var/log/kolla/etcd/etcd.log'}, u'volumes': [u'/etc/kolla//etcd/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'kolla_etcd:/var/lib/etcd/', u'kolla_logs:/var/log/kolla/'], u'container_name': u'etcd'}})

TASK [etcd : Copying over config.json files for services] **********************
changed: [au11-10-wcp] => (item={'key': u'etcd', 'value': {u'group': u'etcd', u'image': u'10.166.30.60:4000/ocata/centos-source-etcd:2.4rc1', u'enabled': True, u'environment': {u'ETCD_ADVERTISE_CLIENT_URLS': u'http://10.11.117.85:2379', u'ETCD_INITIAL_CLUSTER_STATE': u'new', u'ETCD_INITIAL_ADVERTISE_PEER_URLS': u'http://10.11.117.85:2380', u'ETCD_INITIAL_CLUSTER': u'au11-10-wcp=http://10.11.117.85:2380', u'ETCD_DATA_DIR': u'/var/lib/etcd', u'KOLLA_CONFIG_STRATEGY': u'COPY_ALWAYS', u'ETCD_LISTEN_PEER_URLS': u'http://10.11.117.85:2380', u'ETCD_INITIAL_CLUSTER_TOKEN': u'RYpEl4zUZJfGAvMtVcOA9fnloL5CdRENtf0BWlAv', u'ETCD_NAME': u'au11-10-wcp', u'ETCD_LISTEN_CLIENT_URLS': u'http://10.11.117.85:2379', u'ETCD_OUT_FILE': u'/var/log/kolla/etcd/etcd.log'}, u'volumes': [u'/etc/kolla//etcd/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'kolla_etcd:/var/lib/etcd/', u'kolla_logs:/var/log/kolla/'], u'container_name': u'etcd'}})

TASK [etcd : Check etcd containers] ********************************************
changed: [au11-10-wcp] => (item={'key': u'etcd', 'value': {u'group': u'etcd', u'image': u'10.166.30.60:4000/ocata/centos-source-etcd:2.4rc1', u'enabled': True, u'environment': {u'ETCD_ADVERTISE_CLIENT_URLS': u'http://10.11.117.85:2379', u'ETCD_INITIAL_CLUSTER_STATE': u'new', u'ETCD_INITIAL_ADVERTISE_PEER_URLS': u'http://10.11.117.85:2380', u'ETCD_INITIAL_CLUSTER': u'au11-10-wcp=http://10.11.117.85:2380', u'ETCD_DATA_DIR': u'/var/lib/etcd', u'KOLLA_CONFIG_STRATEGY': u'COPY_ALWAYS', u'ETCD_LISTEN_PEER_URLS': u'http://10.11.117.85:2380', u'ETCD_INITIAL_CLUSTER_TOKEN': u'RYpEl4zUZJfGAvMtVcOA9fnloL5CdRENtf0BWlAv', u'ETCD_NAME': u'au11-10-wcp', u'ETCD_LISTEN_CLIENT_URLS': u'http://10.11.117.85:2379', u'ETCD_OUT_FILE': u'/var/log/kolla/etcd/etcd.log'}, u'volumes': [u'/etc/kolla//etcd/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'kolla_etcd:/var/lib/etcd/', u'kolla_logs:/var/log/kolla/'], u'container_name': u'etcd'}})

TASK [etcd : include] **********************************************************
included: /usr/share/kolla-ansible/ansible/roles/etcd/tasks/bootstrap.yml for au11-10-wcp

TASK [etcd : Running etcd bootstrap container] *********************************
changed: [au11-10-wcp -> 10.166.30.136]

[Charlie Kang (charlie-kang)]

Checking /var/lib/docker/containers/{GUID of container}/config.v2.json

Env is not populated with what is set above.

Appears environment is not passed into the containers properly.

[Charlie Kang (charlie-kang)]

Found missing environment param under handlers, submitted gerrit review

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/439937
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=099a1ea17dff083924e6932c69e6d6ec5beaa440
Submitter: Jenkins
Branch: master

commit 099a1ea17dff083924e6932c69e6d6ec5beaa440
Author: Charlie Kang <email address hidden>
Date: Wed Mar 1 15:49:19 2017 -0800

    Etcd fails to start - missing environment parameter

    add environment parameter to handlers restart etcd container as
    there are env parameters in default

    Closes-Bug: #1669089

    Change-Id: I99266e7a367cb54c77b3d3da35ba5055acd98f62

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/440630
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f8c4743ba48974a4501cff2883ebec50eeed27f4
Submitter: Jenkins
Branch: stable/ocata

commit f8c4743ba48974a4501cff2883ebec50eeed27f4
Author: Charlie Kang <email address hidden>
Date: Wed Mar 1 15:49:19 2017 -0800

    Etcd fails to start - missing environment parameter

    add environment parameter to handlers restart etcd container as
    there are env parameters in default

    Closes-Bug: #1669089

    Change-Id: I99266e7a367cb54c77b3d3da35ba5055acd98f62
    (cherry picked from commit 099a1ea17dff083924e6932c69e6d6ec5beaa440)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 4.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.