Failed to change ownership of /var/lib/mistral/undercloud.conf

Bug #1814275 reported by Jose Luis Franco on 2019-02-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Jose Luis Franco

Bug Description

In the tripleo-ci-centos-7-containerized-undercloud-upgrades we can see the following error in the mistral_executor container docker logs:

INFO:__main__:Setting permission for /var/lib/mistral
INFO:__main__:Setting permission for /var/lib/mistral/undercloud.conf
ERROR:__main__:Failed to change ownership of /var/lib/mistral/undercloud.conf to 42430:42430
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/var/lib/mistral/undercloud.conf'
++ cat /run_command
+ CMD='/usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor'

http://logs.openstack.org/30/634330/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/94198d2/logs/undercloud/var/log/extra/docker/containers/mistral_executor/stdout.log.txt.gz

As well as in the journal:

Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: INFO:__main__:Setting permission for /var/lib/mistral/undercloud.conf
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 sudo[66635]: pam_unix(sudo:session): session closed for user root
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: ERROR:__main__:Failed to change ownership of /var/lib/mistral/undercloud.conf to 42430:42430
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: Traceback (most recent call last):
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: os.chown(path, uid, gid)
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: OSError: [Errno 30] Read-only file system: '/var/lib/mistral/undercloud.conf'
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: ++ cat /run_command
Jan 31 23:20:03 centos-7-rax-dfw-0002312926 dockerd-current[19403]: + CMD='/usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor'

http://logs.openstack.org/30/634330/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/94198d2/logs/undercloud/var/log/journal.txt.gz#_Jan_31_23_20_03

Jose Luis Franco (jfrancoa) wrote :

The main issue looks like we're mapping /home/stack/undercloud.conf to /var/lib/mistral/undercloud.conf as read-only: https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/mistral/mistral-executor-container-puppet.yaml#L183

But when running the mistral command, we set the permission field to change the ownership for the whole /var/lib/mistral directory (in a recursive manner) and as undercloud.conf is a read-only file we fail. The easies solution is to just map the undercloud.conf as rw.

Fix proposed to branch: master
Review: https://review.openstack.org/634425

Changed in tripleo:
status: Triaged → In Progress
tags: added: rocky-backport-potential

Reviewed: https://review.openstack.org/634425
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2587f4e961053a6fcca2502d49bc9df0b1f35a97
Submitter: Zuul
Branch: master

commit 2587f4e961053a6fcca2502d49bc9df0b1f35a97
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri Feb 1 15:58:45 2019 +0100

    Copy undercloud.conf file during mistral-executor start up.

    As we are changing the ownership of the whole /var/lib/mistral directory in a
    recursive way, we can't be mapping /var/lib/mistral/undercloud.conf as read-only,
    otherwise we will get a OSError: [Errno 30] Read-only file system: '/var/lib/mistral/undercloud.conf'.

    Instead, we will be mapping the undercloud.conf file into /var/lib/ directory
    inside the container and then copy it from /var/lib into /var/lib/mistral
    directory on start up, this way we will get rid of ownership permission issues.

    Change-Id: Icbdaaf628d996bac89a770426db97d556df4003a
    Closes-Bug: #1814275

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers