iscsid container: mkdir /sys/kernel/config: operation not permitted

Bug #1631072 reported by Waldemar Znoinski on 2016-10-06
62
This bug affects 13 people
Affects Status Importance Assigned to Milestone
kolla
High
Mark Goddard
Stein
High
Mark Goddard

Bug Description

using kolla master + https://review.openstack.org/#/c/381942/2

host/container os ubuntu, source
grep -viE "^#|^$" /etc/kolla/globals.yml
---
config_strategy: "COPY_ALWAYS"
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "3.0.0"
kolla_internal_vip_address: "192.168.203.21"
enable_haproxy: "no"
network_interface: "ens3"
neutron_external_interface: "ens10"
openstack_logging_debug: "True"
enable_ironic: "yes"
enable_iscsid: "yes"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

TASK [iscsi : Starting iscsid container] ***************************************
task path: /kolla_ironic_master/ansible/roles/iscsi/tasks/start.yml:2
ESTABLISH LOCAL CONNECTION FOR USER: root
localhost EXEC ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134 )" )
localhost PUT /tmp/tmplL4zFZ TO /root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker
localhost EXEC LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker; rm -rf "/root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/" > /dev/null 2>&1
fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "invocation": {"module_args": {"action": "start_container", "api_version": "auto", "auth_email": null, "auth_password": null, "auth_registry": null, "auth_username": null, "cap_add": [], "detach": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "KOLLA_SERVICE_NAME": "iscsid"}, "image": "kolla/ubuntu-source-iscsid:3.0.0", "ipc_mode": "host", "labels": {}, "name": "iscsid", "pid_mode": null, "privileged": true, "remove_on_exit": true, "restart_policy": "unless-stopped", "restart_retries": 10, "security_opt": [], "tls_cacert": null, "tls_cert": null, "tls_key": null, "tls_verify": false, "volumes": ["/etc/kolla//iscsid/:/var/lib/kolla/config_files/:ro", "kolla_logs:/var/log/kolla/", "/etc/localtime:/etc/localtime:ro", "/dev/:/dev/", "/run/:/run/", "/sys/fs/cgroup:/sys/fs/cgroup:ro", "/lib/modules:/lib/modules:ro", "/sys/kernel/config:/configfs", "cinder:/var/lib/cinder", "iscsi_info:/etc/iscsi"], "volumes_from": null}, "module_name": "kolla_docker"}, "msg": "'Traceback (most recent call last):\\n File \"/root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker\", line 743, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/root/.ansible/tmp/ansible-tmp-1475771084.12-37262473940134/kolla_docker\", line 573, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/container.py\", line 363, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 146, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"mkdir /sys/kernel/config: operation not permitted\"}\")\\n'"}

there doesn't seem to by /sys/kernel/config directory on host ubuntu xenial

Tags: doc Edit Tag help
Jeffrey Zhang (jeffrey4l) wrote :

/sys/kernel/config this folder is empty on centos, too. it is wired. I will test the iscsi with the latest code.

Mark Duggan (mark-duggan-t) wrote :

/sys/kernel/config does not exist on ol6 also. But does exist on ol7 and is empty.

Jeffrey Zhang (jeffrey4l) wrote :

could u check this[0]. is this service is started on ubuntu?

[0] https://github.com/systemd/systemd/blob/master/units/sys-kernel-config.mount

Steven Dake (sdake) on 2016-10-08
Changed in kolla:
status: New → Triaged
importance: Undecided → High
milestone: none → newton-rc2
Jeffrey Zhang (jeffrey4l) wrote :

The root cause may be the configfs module is not loaded[0]

# systemctl status sys-kernel-config.mount
# modprobe configfs
# systemctl start sys-kernel-config.mount
# systemctl status sys-kernel-config.mount

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800521

Jeffrey Zhang (jeffrey4l) wrote :

If so, we should add the fix to the doc.

Waldemar Znoinski (wznoinsk) wrote :

it's deploying fine after loading configfs into kernel on the host
I didn't seem to need it for ironic with pxe_ipmitool driver, I guess it's a requirement for cinder, worth documenting by someone familiar with the topic

tags: added: docimpact
tags: added: doc
removed: docimpact
Steven Dake (sdake) on 2016-10-13
Changed in kolla:
milestone: newton-rc2 → ocata-1
Changed in kolla:
status: Triaged → Confirmed
Changed in kolla:
milestone: ocata-1 → ocata-2
Changed in kolla:
milestone: ocata-2 → ocata-3
Changed in kolla:
milestone: ocata-3 → ocata-rc1
Changed in kolla:
milestone: ocata-rc1 → pike-1
Changed in kolla:
milestone: pike-2 → pike-3
Changed in kolla:
milestone: pike-3 → pike-rc1
Changed in kolla:
milestone: pike-rc1 → queens-1
Changed in kolla:
milestone: queens-2 → queens-3

Why is this being postponed???

Jeffrey Zhang (jeffrey4l) wrote :

This is a OS bug actually. and the fix solution is mentioned in comment #4

Chason Chan (chen-xing) wrote :

will import this to docs ASAP.

Changed in kolla:
assignee: nobody → Chason Chan (chen-xing)
Jeffrey Zhang (jeffrey4l) wrote :

think this issue again. how about load the configfs module during starting iscsi container? this should work.

Gaëtan Trellu (goldyfruit) wrote :

@Jeffrey it's not the module to load, it seems builtin within the kernel already.
We have to make sure that configfs is mounted within /sys/kernel/config

This is what sys-kernel-config.mount systemd unit does.

[Mount]
What=configfs
Where=/sys/kernel/config
Type=configfs

Changed in kolla:
milestone: queens-3 → queens-rc1
Changed in kolla:
milestone: queens-rc1 → queens-rc2
Changed in kolla:
milestone: queens-rc2 → rocky-1

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (OCATA, PIKE, QUEENS, ROCKY, ROCKY).
  Valid example: CONFIRMED FOR: OCATA

Changed in kolla:
assignee: Chason Chan (chen-xing) → nobody
importance: High → Undecided
status: Confirmed → Expired
Mark Goddard (mgoddard) wrote :
Changed in kolla:
status: Expired → Confirmed
Mark Goddard (mgoddard) wrote :

The ironic CI job review is here: https://review.openstack.org/#/c/568829.

Gaëtan Trellu (goldyfruit) wrote :

@Jeffrey, you were right.
The module needs to be loaded, I guess it depends of the kernel compiled.

Changed in kolla:
milestone: rocky-2 → rocky-3
jaya (jaya.os) wrote :

For ubuntu 16.04 and queens release, configfs needs to be manually added. This should be put in the documentation, as this is critical to have succcesfull kolla-ansible installation.

jaya (jaya.os) wrote :

Error encountered: nova-ssh, nova-libvirt, and nova-compute failed to load (timeout), resulting failed deployment of kolla-ansible.

Sample error log:
FAILED - RETRYING: Waiting for nova-compute service up (3 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (2 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (1 retries left).
fatal: [control01 -> control01]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["docker", "exec", "kolla_toolbox", "openstack", "--os-interface", "internal", "--os-auth-url", "http://x.x.x.x:35357", "--os-identity-api-version", "3", "--os-project-domain-name", "default", "--os-tenant-name", "admin", "--os-username", "admin", "--os-password", "bpduEc6eOACQmRZASjcjWS4Qtz40dSH6", "--os-user-domain-name", "default", "compute", "service", "list", "-f", "json", "--service", "nova-compute"], "delta": "0:00:01.699082", "end": "2018-08-19 02:49:31.570876", "rc": 0, "start": "2018-08-19 02:49:29.871794", "stderr": "", "stderr_lines": [], "stdout": "[]", "stdout_lines": ["[]"]}

Mark Goddard (mgoddard) wrote :
Changed in kolla:
assignee: nobody → Mark Goddard (mgoddard)
importance: Undecided → High

This issue was fixed in the openstack/kolla-ansible 8.0.0.0b1 development milestone.

Mark Goddard (mgoddard) wrote :

Fix merged to master, will be present in the Stein release.

Mark Goddard (mgoddard) wrote :

I should says that the fix was actually made to kolla-ansible. Kernel modules are no longer loaded by kolla images, as of Stein.

Changed in kolla:
milestone: rocky-3 → none

This issue was fixed in the openstack/kolla-ansible 7.1.1 release.

Mark Goddard (mgoddard) on 2019-08-07
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.