Ceph partition on the second drive was formatted to 'ext4' instead of 'xfs' during deployment

[Artem Panchenko]

api: '1.0'
astute_sha: 2b19f9cda281a4e53fb4a2e241d882e29ab9724b
build_id: 2014-05-15_01-10-31
build_number: '205'
fuellib_sha: 1c7d598a65309f8bda1391360f0b94bfb5015180
fuelmain_sha: fc249cd1fe62d5303f531cf00d9e31ee2ab642d2
mirantis: 'yes'
nailgun_sha: 0c88805f28d658be82b305eaeafc5466b563569d
ostf_sha: 69bdb9729cbf16d2c8ba927e946d64a8898909af
production: docker
release: '5.0'

Steps to reproduce:

 1. Create new environment (simple + CentOS + Nova/Flat + Ceph for Volumes and Images)
 2. Add 1 controller, 1 ceph-osd, 1 (compute + ceph-osd)
 3. Deploy cluster
 4. After deployment log in to 'ceph-osd' or 'compute + ceph-osd' node and check types of Filesystem used for partitions on Ceph drives

Expected result:
all partitions use 'xfs' filesystem

Actual result:

the partition on the second drive was formatted to ext4:

[root@node-2 ~]# parted /dev/vdb4 print
Model: Virtio Block Device (virtblk)
Disk /dev/vdb4: 53,0GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number Start End Size File system Flags
 1 0,00B 53,0GB 53,0GB xfs

[root@node-2 ~]# parted /dev/vdc4 print
Model: Virtio Block Device (virtblk)
Disk /dev/vdc4: 53,0GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number Start End Size File system Flags
 1 0,00B 53,0GB 53,0GB ext4

Also, I found that puppet logs contain info about preparing /dev/vdb drive, but there is nothing about /dev/vdc:

<29>May 15 06:41:34 node-2 puppet-user[17139]: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns) [ESC[1mceph_deploy.osdESC[0m][ESC[1;34mDEBUGESC[0m ] Preparing host node-2 disk /dev/vdb4 journal None activate False
<29>May 15 06:41:34 node-2 puppet-user[17139]: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns) [ESC[1mnode-2ESC[0m][ESC[1;37mINFOESC[0m ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/vdb4

[Mike Scherbakov (mihgen)]

Preliminary, looks like only Ubuntu is affected by this.

[Dmitry Ilyin (idv1985)]

I cannot confirm this

df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda3 ext4 14G 1.4G 12G 11% /
udev devtmpfs 991M 12K 991M 1% /dev
tmpfs tmpfs 201M 288K 200M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 1001M 0 1001M 0% /run/shm
/dev/vda2 ext3 185M 58M 119M 33% /boot
/dev/vdb2 xfs 50G 2.1G 48G 5% /var/lib/ceph/osd/ceph-1

As you can see Ceph partition on second drive is formated to xfs

Deployment is Ubuntu HA with two dedicated ceph nodes both with two drives.
ISO: 206

[Artem Panchenko (apanchenko-8)]

Dmitry, in the test environment the nodes had 3 drives: 1 (/dev/vda) for base system and 2 (/dev/vd{b,c}) for Ceph storage. And as you can see in my original post, partition on the first Ceph drive was formatted correctly to XFS. So I think this bug can affect systems that allocate more than 1 drive for Ceph.

[Vladimir Kozhukalov (kozhukalov)]

Attached diagnostic snapshot does not contain ubuntu installations. One of the directories which is called 172.17.42.1 instead of something like node-N.domain.name and contains mixture on logs of several centos installations and there are no errors there and there are no messages about ext4 creating there. puppet log contains log messages of creating xfs file system on /dev/vdb4 for node-2 and node-3.

Original bug report says that this error appears on centos not on ubuntu. Looks like incomplete. Will try to reproduce on Centos.

[Dmitry Ilyin (idv1985)]

No confirmation even with two disks:
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda3 ext4 14G 1.4G 12G 11% /
udev devtmpfs 991M 12K 991M 1% /dev
tmpfs tmpfs 201M 300K 200M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 1001M 0 1001M 0% /run/shm
/dev/vda2 ext3 185M 58M 119M 33% /boot
/dev/vdb2 xfs 50G 2.1G 48G 5% /var/lib/ceph/osd/ceph-0
/dev/vdc2 xfs 50G 2.1G 48G 5% /var/lib/ceph/osd/ceph-1

[Vladimir Kozhukalov (kozhukalov)]

Unfortunately, it is almost impossible to download the fresh iso. Looks like will not be able to reproduce the issue. Could you guys try to reproduce the issue on Centos environment with 209 iso and then make diagnostic snapshot?

[Artem Panchenko (apanchenko-8)]

Vladimir, I have just reproduced it on ISO # 208, snapshot is attached.

[root@nailgun ~]# fuel --fuel-version
api: '1.0'
astute_sha: c0418e1739cceb864fd2c548a7c6b9ad6a46ed86
build_id: 2014-05-16_01-10-31
build_number: '208'
fuellib_sha: 68ef90fdc9fe5a3978f756a96a3fa29d5f1a2929
fuelmain_sha: 861f4410bd07ecf40c1f87785dd0c86de387f2cf
mirantis: 'yes'
nailgun_sha: 89e7224f0b87284f75bf02b956b640768fabe352
ostf_sha: 353f918197ec53a00127fd28b9151f248a2a2d30
production: docker
release: '5.0'

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|--------|------------------|---------|--------------|-------------------|-------------------|---------------|-------
3 | ready | Untitled (5f:c9) | 1 | 10.108.155.5 | 64:00:00:b9:5f:c9 | ceph-osd | | True
2 | ready | Untitled (87:56) | 1 | 10.108.155.4 | 64:e2:3b:87:87:56 | ceph-osd, compute | | True
1 | ready | Untitled (8c:48) | 1 | 10.108.155.3 | 64:82:10:8c:8c:48 | controller | | True

[root@nailgun ~]# ssh node-2 "cat /proc/mounts | grep 'vd[a-z]'" 2>/dev/null
/dev/vdc4 /none ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/vdb4 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,delaylog,noquota 0 0

[root@nailgun ~]# ssh node-3 "cat /proc/mounts | grep 'vd[a-z]'" 2>/dev/null
/dev/vdc4 /none ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/vdb4 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,delaylog,noquota 0 0

[Dmitry Borodaenko (angdraug)]

"/dev/vdc4 /none" -- a very suspicious mount point.

[Artem Panchenko (apanchenko-8)]

Dmitry, yep, I guess it was mounted there by anaconda during CentOS installation, here is the part of its log:

http://paste.openstack.org/show/80631/

[Vladimir Kozhukalov (kozhukalov)]

Artem, great. Thanks a lot.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/93983

[Vladimir Kozhukalov (kozhukalov)]

Artem, please, try to test this patch https://review.openstack.org/93983. You can do this without rebuilding iso. You just need to take pmanager.py from the commit, put it in /usr/lib/python2.6/site-packages/cobbler/pmanager.py and restart cobbler. Hope it fix the issue.

[Vladimir Kozhukalov (kozhukalov)]

I've tested it without deployment just running pmanager.py over data which i found in diagnostic snapshot.

[Artem Panchenko (apanchenko-8)]

Vladimir, sorry for delayed, I've tested your patch on iso # 208 and it works like a charm.

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|--------|------------------|---------|--------------|-------------------|-------------------|---------------|-------
8 | ready | Untitled (5f:c9) | 4 | 10.108.155.3 | 64:00:00:b9:5f:c9 | controller | | True
9 | ready | Untitled (8c:48) | 4 | 10.108.155.4 | 64:82:10:8c:8c:48 | ceph-osd, compute | | True
10 | ready | Untitled (87:56) | 4 | 10.108.155.5 | 64:e2:3b:87:87:56 | ceph-osd | | True

[root@nailgun ~]# ssh node-9 "cat /proc/mounts | grep vd[b,c]" 2>/dev/null
/dev/vdb4 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/vdc4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,delaylog,noquota 0 0

[root@nailgun ~]# ssh node-10 "cat /proc/mounts | grep vd[b,c]" 2>/dev/null
/dev/vdc4 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/vdb4 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,delaylog,noquota 0 0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93983
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=edd02814a6398884ab44109cebe558af7d6eef48
Submitter: Jenkins
Branch: master

commit edd02814a6398884ab44109cebe558af7d6eef48
Author: Vladimir Kozhukalov <email address hidden>
Date: Fri May 16 22:18:51 2014 +0400

    Changed condition in partition method

    Fixes wrong condition introduced by
    https://review.openstack.org/#/c/92849

    Closes-Bug: #1319871
    Change-Id: I437ef4c10826852f21cb3e0d435fab72f46cdec3

[Artem Panchenko (apanchenko-8)]

Verified on ISO # 213

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96857

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96857
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3f91609801087a31d7207f897b9e02e8ae7db744
Submitter: Jenkins
Branch: stable/4.1

commit 3f91609801087a31d7207f897b9e02e8ae7db744
Author: Vladimir Kozhukalov <email address hidden>
Date: Fri May 16 22:18:51 2014 +0400

    Changed condition in partition method

    Fixes wrong condition introduced by
    https://review.openstack.org/#/c/92849

    Closes-Bug: #1319871
    Change-Id: I437ef4c10826852f21cb3e0d435fab72f46cdec3

[Meg McRoberts (dreidellhasa)]

Documented as fixed in 4.1.1 Release Notes