iscsi container fails to start, "no space left on device"

Bug #1671447 reported by kbaegis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Invalid
Undecided
Unassigned

Bug Description

Breaks kolla-ansible deploy:

TASK [iscsi : Starting iscsid container] ***************************************
task path: /usr/local/share/kolla-ansible/ansible/roles/iscsi/tasks/start.yml:2
Using module file /usr/local/share/kolla-ansible/ansible/library/kolla_docker.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176 `" && echo ansible-tmp-1489060670.64-177811876367176="` echo ~/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176 `" ) && sleep 0'
<localhost> PUT /tmp/tmpTKfnKm TO /root/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176/kolla_docker.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176/ /root/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176/kolla_docker.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176/kolla_docker.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1489060670.64-177811876367176/" > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": true,
    "failed": true,
    "invocation": {
        "module_args": {
            "action": "start_container",
            "api_version": "auto",
            "auth_email": null,
            "auth_password": null,
            "auth_registry": "127.0.0.1:5893",
            "auth_username": null,
            "cap_add": [],
            "detach": true,
            "environment": {
                "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"
            },
            "image": "127.0.0.1:5893/kolla/ubuntu-source-iscsid:beta",
            "ipc_mode": "host",
            "labels": {},
            "name": "iscsid",
            "privileged": true,
            "remove_on_exit": true,
            "restart_policy": "unless-stopped",
            "restart_retries": 10,
            "security_opt": [],
            "tls_cacert": null,
            "tls_cert": null,
            "tls_key": null,
            "tls_verify": false,
            "volumes": [
                "/etc/kolla//iscsid/:/var/lib/kolla/config_files/:ro",
                "kolla_logs:/var/log/kolla/",
                "/etc/localtime:/etc/localtime:ro",
                "/dev/:/dev/",
                "/run/:/run/:shared",
                "/sys/fs/cgroup:/sys/fs/cgroup:ro",
                "/lib/modules:/lib/modules:ro",
                "/sys/kernel/config:/configfs",
                "cinder:/var/lib/cinder",
                "iscsi_info:/etc/iscsi"
            ],
            "volumes_from": null
        },
        "module_name": "kolla_docker"
    },
    "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_KOGSUr/ansible_module_kolla_docker.py\", line 781, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_KOGSUr/ansible_module_kolla_docker.py\", line 608, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/container.py\", line 383, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"oci runtime error: container_linux.go:247: starting container process caused \\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/dev/shm\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/aufs/mnt/69194e28e89715e60490e06dd933648336507a7942f06dd2a57945b695b3823b\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/aufs/mnt/69194e28e89715e60490e06dd933648336507a7942f06dd2a57945b695b3823b/dev/shm\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\"no space left on device\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\"\\\\\"\\\\n\"}\")\\n'"
}
 to retry, use: --limit @/usr/local/share/kolla-ansible/ansible/site.retry

PLAY RECAP *********************************************************************
localhost : ok=108 changed=0 unreachable=0 failed=1

Command failed ansible-playbook -i /usr/local/share/kolla-ansible/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/local/share/kolla-ansible/ansible/site.yml --verbose --verbose --verbose

There's definitely space on the device:
/dev/mapper/crucible--vg-root 2751134672 8109544 2603252408 1% /

  PV /dev/sda3 VG crucible-vg lvm2 [2.73 TiB / 0 free]
  PV /dev/sdd VG cinder-hdd-volumes lvm2 [5.46 TiB / 5.46 TiB free]
  PV /dev/sdb VG cinder-hdd-volumes lvm2 [1.82 TiB / 1.82 TiB free]
  PV /dev/sdc VG cinder-hdd-volumes lvm2 [5.46 TiB / 5.46 TiB free]
  PV /dev/nvme0n1 VG cinder-flash-volumes lvm2 [476.94 GiB / 476.94 GiB free]
  Total: 5 [15.93 TiB] / in use: 5 [15.93 TiB] / in no VG: 0 [0 ]

Revision history for this message
kbaegis (kbaegis) wrote :
Download full text (5.2 KiB)

Verified on BTRFS as well. Kolla-ansible at stable/ocata, kolla-build at master. Built ubuntu-source, all images pushed to repo and built cleanly.

RUNNING HANDLER [cinder : Restart cinder-backup container] *********************
Using module file /usr/local/share/kolla-ansible/ansible/library/kolla_docker.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150 `" && echo ansible-tmp-1489092236.17-27738499404150="` echo ~/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150 `" ) && sleep 0'
<localhost> PUT /tmp/tmpBpNXph TO /root/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150/kolla_docker.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150/ /root/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150/kolla_docker.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150/kolla_docker.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1489092236.17-27738499404150/" > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": true,
    "failed": true,
    "invocation": {
        "module_args": {
            "action": "recreate_or_restart_container",
            "api_version": "auto",
            "auth_email": null,
            "auth_password": null,
            "auth_registry": "127.0.0.1:5893",
            "auth_username": null,
            "cap_add": [],
            "detach": true,
            "environment": {
                "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
                "KOLLA_SERVICE_NAME": "cinder-backup"
            },
            "image": "127.0.0.1:5893/kolla/ubuntu-source-cinder-backup:zerolabs",
            "labels": {},
            "name": "cinder_backup",
            "privileged": true,
            "remove_on_exit": true,
            "restart_policy": "unless-stopped",
            "restart_retries": 10,
            "security_opt": [],
            "tls_cacert": null,
            "tls_cert": null,
            "tls_key": null,
            "tls_verify": false,
            "volumes": [
                "/etc/kolla//cinder-backup/:/var/lib/kolla/config_files/:ro",
                "/etc/localtime:/etc/localtime:ro",
                "/dev/:/dev/",
                "/run/:/run/:shared",
                "cinder:/var/lib/cinder",
                "iscsi_info:/etc/iscsi",
                "kolla_logs:/var/log/kolla/"
            ],
            "volumes_from": null
        },
        "module_name": "kolla_docker"
    },
    "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_UcgklN/ansible_module_kolla_docker.py\", line 781, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_UcgklN/ansible_module_kolla_docker.py\", line 583, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_UcgklN/ansible_module_kolla_docker.py\", line 608, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorat...

Read more...

Revision history for this message
kbaegis (kbaegis) wrote :

cinder_volume_group: "cinder-volumes" fixes this bug with btrfs.

Looks like a product defect for the cinder_volume_group configuration parameter in globals.

Revision history for this message
Eduardo Gonzalez (egonzalez90) wrote :

As per your comment see the issue was fixed, i'm going to close the bug, is something else is needed please re-open again.
Regards

Changed in kolla:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.