ceph : Bootstrapping Ceph OSDs failed

[rajagopal]

failed: [node3] => (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdc', u'journal': u'/dev/sdc1', u'partition': u'/dev/sdb1', u'partition_num': u'1', u'journal_num': u'1', u'fs_label': u'', u'device': u'/dev/sdb', u'external_journal': True})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "external_journal": true, "fs_label": "", "fs_uuid": "", "journal": "/dev/sdc1", "journal_device": "/dev/sdc", "journal_num": "1", "partition": "/dev/sdb1", "partition_num": "1"}], "msg": "'Traceback (most recent call last):\\n File \"/root/.ansible/tmp/ansible-tmp-1482688181.33-258033820924072/kolla_docker\", line 743, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/root/.ansible/tmp/ansible-tmp-1482688181.33-258033820924072/kolla_docker\", line 573, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/container.py\", line 383, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"rpc error: code = 2 desc = \"oci runtime error: could not synchronise with container process: evalSymlinksInScope: too many links in /var/lib/docker/aufs/mnt/d33c725c8eb02109032ca95fc14f16f4966dbd3489faec68d0dee32a37a81640/dev/shm\"\")\\n'"}

failed: [node2] => (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdc', u'journal': u'/dev/sdc1', u'partition': u'/dev/sdb1', u'partition_num': u'1', u'journal_num': u'1', u'fs_label': u'', u'device': u'/dev/sdb', u'external_journal': True})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "external_journal": true, "fs_label": "", "fs_uuid": "", "journal": "/dev/sdc1", "journal_device": "/dev/sdc", "journal_num": "1", "partition": "/dev/sdb1", "partition_num": "1"}], "msg": "'Traceback (most recent call last):\\n File \"/root/.ansible/tmp/ansible-tmp-1482688181.51-222167753438813/kolla_docker\", line 743, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/root/.ansible/tmp/ansible-tmp-1482688181.51-222167753438813/kolla_docker\", line 573, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/container.py\", line 383, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"rpc error: code = 2 desc = \"oci runtime error: could not synchronise with container process: evalSymlinksInScope: too many links in /var/lib/docker/aufs/mnt/7484245f6467c0eb142bc31eaaaf8a067dadb284ee72e89f57da1a3cb8042893/dev/shm\"\")\\n'"}

failed: [node1] => (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdc', u'journal': u'/dev/sdc1', u'partition': u'/dev/sdb1', u'partition_num': u'1', u'journal_num': u'1', u'fs_label': u'', u'device': u'/dev/sdb', u'external_journal': True})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "external_journal": true, "fs_label": "", "fs_uuid": "", "journal": "/dev/sdc1", "journal_device": "/dev/sdc", "journal_num": "1", "partition": "/dev/sdb1", "partition_num": "1"}], "msg": "'Traceback (most recent call last):\\n File \"/root/.ansible/tmp/ansible-tmp-1482688181.72-223477135713697/kolla_docker\", line 743, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/root/.ansible/tmp/ansible-tmp-1482688181.72-223477135713697/kolla_docker\", line 573, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/container.py\", line 383, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"rpc error: code = 2 desc = \"oci runtime error: could not synchronise with container process: evalSymlinksInScope: too many links in /var/lib/docker/aufs/mnt/656a253a0bab20f489b359065821915df720879e8eebe9ec32762ff9493a4218/dev/shm\"\")\\n'"}

[Jeffrey Zhang (jeffrey4l)]

check your filesystem inode usage. seems docker create many files.

solutions:

1. remove unused / orphan / untagged images.
2. try other docker storage driver, like btrfs.

[rajagopal (rajagopalx)]

I am using ubuntu images.

I got working with stable/mitaka (Kolla Ubuntu Images Tag: 2.0.2). But I got the above error in Newton Release. (Kolla Images Tag: 3.0.0).

I only have required docker images.
I will try with btrfs driver.

[Launchpad Janitor (janitor)]

[Expired for kolla because there has been no activity for 60 days.]