iscsid container: mkdir /sys/kernel/config: operation not permitted

[Waldemar Znoinski]

using kolla master + https://review.openstack.org/#/c/381942/2

host/container os ubuntu, source
grep -viE "^#|^$" /etc/kolla/globals.yml
---
config_strategy: "COPY_ALWAYS"
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "3.0.0"
kolla_internal_vip_address: "192.168.203.21"
enable_haproxy: "no"
network_interface: "ens3"
neutron_external_interface: "ens10"
openstack_logging_debug: "True"
enable_ironic: "yes"
enable_iscsid: "yes"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

TASK [iscsi : Starting iscsid container] ***************************************
task path: /kolla_ironic_master/ansible/roles/iscsi/tasks/start.yml:2
ESTABLISH LOCAL CONNECTION FOR USER: root
localhost EXEC ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134 )" )
localhost PUT /tmp/tmplL4zFZ TO /root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker
localhost EXEC LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker; rm -rf "/root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/" > /dev/null 2>&1
fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "invocation": {"module_args": {"action": "start_container", "api_version": "auto", "auth_email": null, "auth_password": null, "auth_registry": null, "auth_username": null, "cap_add": [], "detach": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "KOLLA_SERVICE_NAME": "iscsid"}, "image": "kolla/ubuntu-source-iscsid:3.0.0", "ipc_mode": "host", "labels": {}, "name": "iscsid", "pid_mode": null, "privileged": true, "remove_on_exit": true, "restart_policy": "unless-stopped", "restart_retries": 10, "security_opt": [], "tls_cacert": null, "tls_cert": null, "tls_key": null, "tls_verify": false, "volumes": ["/etc/kolla//iscsid/:/var/lib/kolla/config_files/:ro", "kolla_logs:/var/log/kolla/", "/etc/localtime:/etc/localtime:ro", "/dev/:/dev/", "/run/:/run/", "/sys/fs/cgroup:/sys/fs/cgroup:ro", "/lib/modules:/lib/modules:ro", "/sys/kernel/config:/configfs", "cinder:/var/lib/cinder", "iscsi_info:/etc/iscsi"], "volumes_from": null}, "module_name": "kolla_docker"}, "msg": "'Traceback (most recent call last):\\n File \"/root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker\", line 743, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker\", line 573, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/container.py\", line 363, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 146, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"mkdir /sys/kernel/config: operation not permitted\"}\")\\n'"}

there doesn't seem to by /sys/kernel/config directory on host ubuntu xenial

[Jeffrey Zhang (jeffrey4l)]

/sys/kernel/config this folder is empty on centos, too. it is wired. I will test the iscsi with the latest code.

[Mark Duggan (mark-duggan-t)]

/sys/kernel/config does not exist on ol6 also. But does exist on ol7 and is empty.

[Jeffrey Zhang (jeffrey4l)]

could u check this[0]. is this service is started on ubuntu?

[0] https://github.com/systemd/systemd/blob/master/units/sys-kernel-config.mount

[Jeffrey Zhang (jeffrey4l)]

The root cause may be the configfs module is not loaded[0]

# systemctl status sys-kernel-config.mount
# modprobe configfs
# systemctl start sys-kernel-config.mount
# systemctl status sys-kernel-config.mount

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800521

[Jeffrey Zhang (jeffrey4l)]

If so, we should add the fix to the doc.

[Waldemar Znoinski (wznoinsk)]

it's deploying fine after loading configfs into kernel on the host
I didn't seem to need it for ironic with pxe_ipmitool driver, I guess it's a requirement for cinder, worth documenting by someone familiar with the topic

[Ravi Shekhar Jethani (ravishekar-jethani)]

Why is this being postponed???

[Jeffrey Zhang (jeffrey4l)]

This is a OS bug actually. and the fix solution is mentioned in comment #4

[Chason Chan (chen-xing)]

will import this to docs ASAP.

[Jeffrey Zhang (jeffrey4l)]

think this issue again. how about load the configfs module during starting iscsi container? this should work.

[Gaëtan Trellu (goldyfruit)]

@Jeffrey it's not the module to load, it seems builtin within the kernel already.
We have to make sure that configfs is mounted within /sys/kernel/config

This is what sys-kernel-config.mount systemd unit does.

[Mount]
What=configfs
Where=/sys/kernel/config
Type=configfs

[Jeffrey Zhang (jeffrey4l)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (OCATA, PIKE, QUEENS, ROCKY, ROCKY).
  Valid example: CONFIRMED FOR: OCATA

[Mark Goddard (mgoddard)]

This is still an issue in master, as can be seen in my WIP ironic CI job: http://logs.openstack.org/29/568829/6/experimental/kolla-ansible-ubuntu-source-ironic/5d2a93b/primary/logs/ansible/deploy.

[Mark Goddard (mgoddard)]

The ironic CI job review is here: https://review.openstack.org/#/c/568829.

[Gaëtan Trellu (goldyfruit)]

@Jeffrey, you were right.
The module needs to be loaded, I guess it depends of the kernel compiled.

[jaya (jaya.os)]

For ubuntu 16.04 and queens release, configfs needs to be manually added. This should be put in the documentation, as this is critical to have succcesfull kolla-ansible installation.

[jaya (jaya.os)]

Error encountered: nova-ssh, nova-libvirt, and nova-compute failed to load (timeout), resulting failed deployment of kolla-ansible.

Sample error log:
FAILED - RETRYING: Waiting for nova-compute service up (3 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (2 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (1 retries left).
fatal: [control01 -> control01]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["docker", "exec", "kolla_toolbox", "openstack", "--os-interface", "internal", "--os-auth-url", "http://x.x.x.x:35357", "--os-identity-api-version", "3", "--os-project-domain-name", "default", "--os-tenant-name", "admin", "--os-username", "admin", "--os-password", "bpduEc6eOACQmRZASjcjWS4Qtz40dSH6", "--os-user-domain-name", "default", "compute", "service", "list", "-f", "json", "--service", "nova-compute"], "delta": "0:00:01.699082", "end": "2018-08-19 02:49:31.570876", "rc": 0, "start": "2018-08-19 02:49:29.871794", "stderr": "", "stderr_lines": [], "stdout": "[]", "stdout_lines": ["[]"]}

[Mark Goddard (mgoddard)]

Fix proposed: https://review.openstack.org/#/c/619026/1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.0.0.0b1 development milestone.

[Mark Goddard (mgoddard)]

Fix merged to master, will be present in the Stein release.

[Mark Goddard (mgoddard)]

I should says that the fix was actually made to kolla-ansible. Kernel modules are no longer loaded by kolla images, as of Stein.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.1.1 release.