hash(datetime.datetime(...)) fails with python3.5 on armhf (on an arm64 host) with a bus error

[Matthias Klose]

seen with
https://launchpad.net/ubuntu/+source/python-cryptography/1.4-2/+build/10476934
https://launchpad.net/ubuntu/+source/python-cryptography/1.5-1/+build/10678695

the TestInvalidityDate.test_invalid_invalidity_date test is the first one to fail with a bus error. No reasonable traceback from gdb.

Trying to build python3.5, python-cffi and python-cryptography with -mno-unaligned-access doesn't fix the issue (buildds running on a 64bit kernel).

[James Page (james-page)]

A little more detail:

../../../tests/test_x509_ext.py::TestInvalidityDate::test_invalid_invalidity_date PASSED
../../../tests/test_x509_ext.py::TestInvalidityDate::test_eq PASSED
../../../tests/test_x509_ext.py::TestInvalidityDate::test_ne PASSED
../../../tests/test_x509_ext.py::TestInvalidityDate::test_repr PASSED
Bus error (core dumped)
E: pybuild pybuild:276: test: plugin distutils failed with: exit code=135: cd /«PKGBUILDDIR»/.pybuild/pythonX.Y_3.5/build; python3.5 -m pytest -v {dir}/tests/
dh_auto_test: pybuild --test -i python{version} -p 3.5 returned exit code 13
make: *** [build-arch] Error 25
../../../tests/test_x509_ext.py::TestInvalidityDate::test_hash debian/rules:18: recipe for target 'build-arch' failed
dpkg-buildpackage: error: debian/rules build-arch gave error exit status 2

[James Page (james-page)]

This should reproduce the bus error:

```
import datetime

from cryptography import x509

def test_hash():
    invalid1 = x509.InvalidityDate(datetime.datetime(2015, 1, 1, 1, 1))
    invalid2 = x509.InvalidityDate(datetime.datetime(2015, 1, 1, 1, 1))
    invalid3 = x509.InvalidityDate(datetime.datetime(2015, 1, 1, 1, 2))
    assert hash(invalid1) == hash(invalid2)
    assert hash(invalid1) != hash(invalid3)

test_hash()
```

but right now every porter I access has a broken dpkg lock preventing me from installing python3-cryptography

[James Page (james-page)]

backtrace from bus error on arm64 host with armhf schroot.

[James Page (james-page)]

backtrace from bus error on arm64 host with armhf schroot.

[James Page (james-page)]

Note that this only occurs when running in an armhf schroot on a arm64 host; running on a armhf host the tests pass OK (including the reproducer).

[James Page (james-page)]

[17328643.359809] Unhandled fault: alignment fault (0x92000021) at 0x00000000f748947d

[James Page (james-page)]

[17328709.319625] python3[7312]: syscall 384
[17328709.319634] Code: 4607b4f0 46114608 e89c461a df000078 (f510bcf0)
[17328709.319639] CPU: 0 PID: 7312 Comm: python3 Not tainted 3.13.0-77-generic #121-Ubuntu
[17328709.319643] task: ffffffc3a4c18b00 ti: ffffffc009398000 task.ti: ffffffc009398000
[17328709.319650] PC is at 0xf7631ea2
[17328709.319652] LR is at 0xdbd9b
[17328709.319654] pc : [<00000000f7631ea2>] lr : [<00000000000dbd9b>] pstate: 800c0030
[17328709.319656] sp : 00000000fffef410
[17328709.319658] x12: 00000000fffef420
[17328709.319661] x11: 000000000030faa8 x10: 0000000000000cd4
[17328709.319664] x9 : 0000000000000000 x8 : 00000000f77eb170
[17328709.319667] x7 : 0000000000000180 x6 : 0000000000331008
[17328709.319669] x5 : 0000000000331008 x4 : 0000000052a7a000
[17328709.319672] x3 : 00000000fffef470 x2 : 0000000000000001
[17328709.319674] x1 : 0000000000000018 x0 : 0000000000330ca0

[17328709.909570] Unhandled fault: alignment fault (0x92000021) at 0x00000000f748947d

[James Page (james-page)]

Dump of assembler code for function siphash24:
   0x000d40a0 <+0>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}
   0x000d40a4 <+4>: movw r3, #64168 ; 0xfaa8
   0x000d40a8 <+8>: sub sp, #156 ; 0x9c
   0x000d40aa <+10>: movt r3, #48 ; 0x30
   0x000d40ae <+14>: movw r4, #3232 ; 0xca0
   0x000d40b2 <+18>: movt r4, #51 ; 0x33
   0x000d40b6 <+22>: mov lr, r1
   0x000d40b8 <+24>: str r3, [sp, #124] ; 0x7c
   0x000d40ba <+26>: ldrd r10, r11, [r4]
   0x000d40be <+30>: cmp.w lr, #7
   0x000d40c2 <+34>: ldr r1, [r3, #0]
   0x000d40c4 <+36>: mov.w r3, lr, lsl #24
   0x000d40c8 <+40>: str r3, [sp, #116] ; 0x74
   0x000d40ca <+42>: addw r3, pc, #1564 ; 0x61c
   0x000d40ce <+46>: ldrd r2, r3, [r3]
   0x000d40d2 <+50>: ldrd r8, r9, [r4, #8]
   0x000d40d6 <+54>: eor.w r5, r11, r3
   0x000d40da <+58>: eor.w r4, r10, r2
   0x000d40de <+62>: addw r7, pc, #1552 ; 0x610
   0x000d40e2 <+66>: ldrd r6, r7, [r7]
   0x000d40e6 <+70>: mov.w r3, #0
   0x000d40ea <+74>: strd r4, r5, [sp]
   0x000d40ee <+78>: str r3, [sp, #112] ; 0x70
   0x000d40f0 <+80>: eor.w r6, r6, r10
   0x000d40f4 <+84>: addw r5, pc, #1536 ; 0x600
   0x000d40f8 <+88>: ldrd r4, r5, [r5]
   0x000d40fc <+92>: eor.w r7, r7, r11
   0x000d4100 <+96>: addw r3, pc, #1532 ; 0x5fc
   0x000d4104 <+100>: ldrd r2, r3, [r3]
   0x000d4108 <+104>: str r0, [sp, #128] ; 0x80
   0x000d410a <+106>: eor.w r4, r4, r8
   0x000d410e <+110>: str r1, [sp, #148] ; 0x94
   0x000d4110 <+112>: eor.w r5, r5, r9
   0x000d4114 <+116>: eor.w r2, r2, r8
   0x000d4118 <+120>: eor.w r3, r3, r9
   0x000d411c <+124>: ble.w 0xd42be <siphash24+542>
   0x000d4120 <+128>: sub.w r1, lr, #8
   0x000d4124 <+132>: str r1, [sp, #120] ; 0x78
   0x000d4126 <+134>: mov r12, r0
   0x000d4128 <+136>: lsrs r1, r1, #3
   0x000d412a <+138>: str r1, [sp, #132] ; 0x84
   0x000d412c <+140>: add.w r10, r1, #1
   0x000d4130 <+144>: add.w r10, r0, r10, lsl #3
   0x000d4134 <+148>: str.w r10, [sp, #108] ; 0x6c
=> 0x000d4138 <+152>: ldrd r10, r11, [r12], #8

[James Page (james-page)]

(gdb) info registers
r0 0xf748947d 4148728957
r1 0x0 0
r2 0xea5e42a5 3932045989
r3 0x553d1b9f 1430068127
r4 0xfd4e48bb 4249766075
r5 0x45370d9c 1161235868
r6 0xe11b8913 3776678163
r7 0x5404df32 1409605426
r8 0x932a27d6 2469013462
r9 0x21587ffd 559448061
r10 0xf7489485 4148728965
r11 0x276bb257 661369431
r12 0xf748947d 4148728957
sp 0xfffeee00 0xfffeee00
lr 0xa 10
pc 0xd4138 0xd4138 <siphash24+152>
cpsr 0x400e0030 1074659376

[James Page (james-page)]

Even simpler reproducer:

>>> import datetime
>>> print(hash(datetime.datetime(2015, 1, 1, 1, 1)))
Bus error (core dumped)

[Matthias Klose (doko)]

with a -O0 build:

Program received signal SIGBUS, Bus error.
0x001a761e in siphash24 (src=0xf74e6fcd, src_sz=10) at ../Python/pyhash.c:384
384 uint64_t mi = _le64toh(*in);
(gdb) info locals
mi = 12884829592
k0 = 2790264422508071935
k1 = 4214380634785109344
b = 720575940379279360
in = 0xf74e6fcd
v0 = 6185254077475221130
v1 = 6778778058880104973
v2 = 5386417098822285726
v3 = 5627556395274645523
t = 1
pt = 0xf74b0320 ""
m = 0x1 <error: Cannot access memory at address 0x1>
(gdb) info args
src = 0xf74e6fcd
src_sz = 10

where _le64toh is defined as
# define _le64toh(x) ((uint64_t)(x))

[Matthias Klose (doko)]

no difference with gcc-5/gcc-6.

[Matthias Klose (doko)]

>>> int(0xf74e6fcd)
4149112781

so indeed an unaligned pointer.

[Matthias Klose (doko)]

the datetime type is defined as

#define _PyTZINFO_HEAD \
    PyObject_HEAD \
    Py_hash_t hashcode; \
    char hastzinfo; /* boolean flag */

typedef struct
{
    _PyTZINFO_HEAD
    unsigned char data[_PyDateTime_DATE_DATASIZE];
} PyDateTime_Date;

and data is used to calculate the hash of the object, not being 4 byte aligned, you get the bus error. Inserting three fill bytes, are making the data member 4-byte aligned solves the issue, however introducing an ABI change makes the new datetime ABI incompatible. For now I don't know a solution to make that change and keep the ABI compatible.

[Matthias Klose (doko)]

python-cryptography now successfully built again

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.5 - 3.5.2-4ubuntu2

---------------
python3.5 (3.5.2-4ubuntu2) yakkety; urgency=low

  * Disable the Pgen cache tests; not shipping the pickled grammar files.

 -- Matthias Klose <email address hidden> Sat, 10 Sep 2016 10:43:12 +0200

[Chris J Arges (arges)]

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

Hello Matthias, or anyone else affected,

Accepted python3.5 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.5/3.5.2-2~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Matthias Klose (doko)]

the updated binaries don't crash with bus errors anymore in xenial-proposed and trusty-proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4.3-1ubuntu1~14.04.4

---------------
python3.4 (3.4.3-1ubuntu1~14.04.4) trusty-proposed; urgency=medium

  * SRU: LP: #1620754: Fix invalid code in pyhash/siphash24. Issue #28055.

 -- Matthias Klose <email address hidden> Sat, 10 Sep 2016 10:26:44 +0200

[Martin Pitt (pitti)]

The verification of the Stable Release Update for python3.4 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.5 - 3.5.2-2~16.04

---------------
python3.5 (3.5.2-2~16.04) xenial-proposed; urgency=medium

  * SRU: LP: #1620754: Fix invalid code in pyhash/siphash24. Issue #28055.

 -- Matthias Klose <email address hidden> Sat, 10 Sep 2016 10:21:44 +0200