Different erasure codes metadata checksums with pypy3 and python2

[Andrey Groshrev]

I wanted to try running swift on pypy.
On pypy2 crashes with a similar error:
https://bitbucket.org/pypy/pypy/issues/2969/attributeerror-threadrlock-object-has-no
On pypy3 running (from snapd).
I started with proxy service.

But it turned out that I can download the file, but I can not download it.
When I upload a file through pypy3 proxy, the code "201" is returned to me. Chunks of the file are saved to disks, but when I try to download its object servers they return the code "500", and a message is visible in the logs
"liberasurecode [...]: Invalid fragment header information!"
And they are quarantined.

This is bad enough because I believe that if I wrote it down, then in any case I will receive my data. In fact, metadata is validated while reading.

When I upload a file through the usual proxy "python2/CentOS 7/Train", I can not read it through with the pypy3 proxy, because Now he falls, because Invalid header checksum.

I compared both versions of the file — they differ only in the header metadata checksum!

00000000 03 00 00 00 00 00 08 00 00 00 00 00 00 00 20 00 |.............. .|
00000010 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 06 00 00 01 00 cc 5e 0c 0b 00 |............^...|
00000040 05 01 00 ed 64 7d 96 00 00 00 00 00 00 00 00 00 |....d}..........|

00000000 03 00 00 00 00 00 08 00 00 00 00 00 00 00 20 00 |.............. .|
00000010 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 06 00 00 01 00 cc 5e 0c 0b 00 |............^...|
00000040 05 01 00 23 e8 12 7b 00 00 00 00 00 00 00 00 00 |...#..{.........|

In addition, I found https://bugs.launchpad.net/liberasurecode/+bug/1666320
And I checked the metadata of my existing clusters with the attached script.
According to his version, they are all not correct!
But they are assembled by their regular packages RH!

[Tim Burke (1-tim-z)]

What version of liberasurecode are you using? From the frag header dumps, it looks like 1.5.0, is that right? It looks like the fix for #1666320 (https://github.com/openstack/liberasurecode/commit/a9b20ae6a) first appeared in 1.6.0 -- if you're on an older version, can you try upgrading and see if that resolves the issue?

FWIW, if you for some reason *can't* upgrade but you *do* at least have some control over the env vars set when running swift, you should be able to use LD_PRELOAD to ensure all services are using the zlib crc. I'd strongly recommend upgrading though (potentially even building from source rather than using system packages).

[Andrey Groshrev (greenx)]

I understood you.
In my opinion, I tried both the old version and the new one.
Now I’ll fix the file system and try again. (For some reason, XFS broke down on several disks when I install liberasure 1.6.1 from the source code. I’m not sure if this correlates, but I have to fix it first).

In any case, it turns out that you cannot use different versions of liberasure in the same cluster and you cannot update the cluster without stopping.

[Tim Burke (1-tim-z)]

Wrote up patches to liberasurecode (https://review.opendev.org/#/c/738959/) and swift (https://review.opendev.org/#/c/739164/) -- I should've remembered this bug report before writing up https://bugs.launchpad.net/swift/+bug/1886088 :-/

THANK YOU for the report! Sorry it's taken a bit to address.