O_DIRECT writes not works for XFS on disks with 4K sectors

[Roman Sokolkov]

Description:
File injection fails with http://paste.openstack.org/show/79000/ in /var/log/nova-all.log

Environment:
- Fuel 4.1A
- HA + Neutron GRE
- CentOS
- Cisco C240 M3S servers + LSI Logic / Symbios Logic MegaRAID SAS 2208

Steps to reproduce:

- Use disk with 4K sectors for "Virtual Storage"(/var/lib/nova).
- Deploy Cluster on CentOS (Simple or HA no matter)
- try to create VM with file injection: nova boot --flavor 1 --image TestVM --file file=openc VM1

Expected result:
- VM status ACTIVE

Actual result:
- VM status ERROR

Details:
Disk has logical sector=512b and physical=4096b
XFS uses sector size = 4096 bytes .
And unfortunately in that case XFS unable to provide O_DIRECT writes. (that are used by guestfs - "cache=none" option)
I've found fresh patch in mailing lists that works around this issue http://oss.sgi.com/archives/xfs/2014-01/msg00255.html .

To check issue:

guestfish -a /var/lib/nova/instances/627a3ce2-0f4a-4ba3-9f4b-2859861fbef1/disk -i

libguestfs: error: guestfs_launch failed.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
and/or run 'libguestfs-test-tool'.

or

dd if=/var/lib/nova/instances/627a3ce2-0f4a-4ba3-9f4b-2859861fbef1/disk of=/dev/null iflag=direct
dd: reading `/var/lib/nova/instances/627a3ce2-0f4a-4ba3-9f4b-2859861fbef1/disk': Invalid argument
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000131719 s, 0.0 kB/s

# parted /dev/sdj print
Model: LSI MR9271-8i (scsi)
Disk /dev/sdj: 1797GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 17.4kB 25.2MB 25.1MB primary bios_grub
 2 25.2MB 235MB 210MB primary boot
 3 235MB 445MB 210MB ext2 primary boot
 4 445MB 1797GB 1796GB primary lvm

# xfs_info /var/lib/nova/
meta-data=/dev/mapper/vm-nova isize=256 agcount=32, agsize=13705216 blks
         = sectsz=4096 attr=2, projid32bit=0
data = bsize=4096 blocks=438566912, imaxpct=5
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=214144, version=2
         = sectsz=4096 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

Workaround:

On live system:
- recreate XFS with sector size=512

/etc/init.d/openstack-nova-compute stop
cd /var/lib/
cp -a nova nova2
umount nova
mkfs.xfs -f -s size=512 /dev/vm/nova
mount -a
cp -a nova2/* nova
/etc/init.d/openstack-nova-compute start

In fuel:
use ext4 for "virtual storage"

In nailgun/fixtures/openstack.yaml set file_system: "ext4" for virtual storage volume.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/92849

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92849
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5128535954a7aa1c456fc3ae31396141660f0383
Submitter: Jenkins
Branch: master

commit 5128535954a7aa1c456fc3ae31396141660f0383
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 8 18:45:53 2014 +0400

    Set xfs sector size into 512

    By default mkfs.xfs sets sector size equal to
    physical block size which makes it impossible to have
    O_DIRECT mode. Enforcing xfs to use 512 byte size
    fixes this problem but decreases fs performance a bit.

    Change-Id: I4b38ef148c16baacfba12ed07534a3761b3cecf1
    Related-Bug: #1316266

[Roman Sokolkov (rsokolkov)]

I'm using latest 5.0 iso and noticed that anaconda failed to create XFS filesystem

post-partiton.log
mkfs.xfs -f -s 512 /dev/mapper/image-glance
+ mkfs.xfs -f -s 512 /dev/mapper/image-glance
unknown option -s 512
....

Options should be:

"-f -s size=512"

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/93535

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93535
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=358ead05d6743ac5cbdb5437868d45e9a6707a97
Submitter: Jenkins
Branch: master

commit 358ead05d6743ac5cbdb5437868d45e9a6707a97
Author: Mike Scherbakov <email address hidden>
Date: Wed May 14 09:54:37 2014 +0400

    Fixed incorrect params to mkfs.xfs

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Ifd51891d18d2c61724b169785cc59debd52b4db0

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/93726

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93726
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=96a614f0a0ed85482cb47bc3491d8699736b74e8
Submitter: Jenkins
Branch: master

commit 96a614f0a0ed85482cb47bc3491d8699736b74e8
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 15 17:37:11 2014 +0400

    Fixed incorrect params to mkfs.xfs for ubuntu

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Id67ab7f1d47d62c3e3d82a4bca4fa538132d25db

[Meg McRoberts (dreidellhasa)]

Added note to "Fixed Issues" in 5.0 Release Notes.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96851

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96855

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96856

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96851
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=66825b50f279e281c4d2ef1c2cb4899a1c47a085
Submitter: Jenkins
Branch: stable/4.1

commit 66825b50f279e281c4d2ef1c2cb4899a1c47a085
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 8 18:45:53 2014 +0400

    Set xfs sector size into 512

    By default mkfs.xfs sets sector size equal to
    physical block size which makes it impossible to have
    O_DIRECT mode. Enforcing xfs to use 512 byte size
    fixes this problem but decreases fs performance a bit.

    Change-Id: I4b38ef148c16baacfba12ed07534a3761b3cecf1
    Related-Bug: #1316266

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96855
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=94a06e8d64d33de6c50cc18cf17ba66cb692669e
Submitter: Jenkins
Branch: stable/4.1

commit 94a06e8d64d33de6c50cc18cf17ba66cb692669e
Author: Mike Scherbakov <email address hidden>
Date: Wed May 14 09:54:37 2014 +0400

    Fixed incorrect params to mkfs.xfs

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Ifd51891d18d2c61724b169785cc59debd52b4db0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96856
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=270bf259e3e109fbb8921e6c2cc992e1103fb06a
Submitter: Jenkins
Branch: stable/4.1

commit 270bf259e3e109fbb8921e6c2cc992e1103fb06a
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu May 15 17:37:11 2014 +0400

    Fixed incorrect params to mkfs.xfs for ubuntu

    Thanks to Roman Sokolkov for regression report.
    Related-Bug: #1316266

    Change-Id: Id67ab7f1d47d62c3e3d82a4bca4fa538132d25db

[Meg McRoberts (dreidellhasa)]

Documented as fixed in 5.0 and 4.1.1 Release Notes