Comment 2 for bug 1882535

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

It looks like my workaround doesn't quite work. And the reason is not snapcraft which correctly passes the environment variable down.

I found that pip is also not a problem. I was using a simple package installation as a test case and found that py_compile correctly gets the SOURCE_DATE_EPOCH variable and decides that SOURCE_DATE_EPOCH method should be used.
https://paste.ubuntu.com/p/63d5wD3DsZ/

So, in the end, we correctly get to this point where the .pyc content is generated:
https://github.com/python/cpython/blob/0f5a28f834bdac2da8a04597dc0fc5b71e50da9d/Lib/py_compile.py#L164-L172

However, the importlib._bootstrap_external._code_to_hash_pyc function returns a different value for the same source during different executions (change of a process):

root@ftest:~/test# rm -r ./*
root@ftest:~/test# pip3 install -t ./ setuptools

> /usr/lib/python3.8/py_compile.py(169)compile()
-> bytecode = importlib._bootstrap_external._code_to_hash_pyc(
(Pdb) print(invalidation_mode) ; import hashlib ; hashlib.sha256(source_bytes).hexdigest() ; hashlib.sha256(source_hash).hexdigest() ; hashlib.sha256(importlib._bootstrap_external._code_to_hash_pyc(code, source_hash, (invalidation_mode == PycInvalidationMode.CHECKED_HASH))).hexdigest() ; importlib._bootstrap_external.MAGIC_NUMBER
PycInvalidationMode.CHECKED_HASH
'a33213258b106b1cedbec418662b29f4226a91e9f579dfc4218722f2a826a2b5'
'6750d9e9657c3ce3ad48843f9d615ece9310755cbfb9deb037c3c9ce2d5e249f'
'4e537d6107457ed2e875030d2717961f319c4da55b0d012fcb09f38d91eda6cd'
b'U\r\r\n'

root@ftest:~/test# rm -r ./*
root@ftest:~/test# pip3 install -t ./ setuptools

(Pdb) print(invalidation_mode) ; import hashlib ; hashlib.sha256(source_bytes).hexdigest() ; hashlib.sha256(source_hash).hexdigest() ; hashlib.sha256(importlib._bootstrap_external._code_to_hash_pyc(code, source_hash, (invalidation_mode == PycInvalidationMode.CHECKED_HASH))).hexdigest() ; importlib._bootstrap_external.MAGIC_NUMBER
PycInvalidationMode.CHECKED_HASH
'a33213258b106b1cedbec418662b29f4226a91e9f579dfc4218722f2a826a2b5'
'6750d9e9657c3ce3ad48843f9d615ece9310755cbfb9deb037c3c9ce2d5e249f'
'8d7d7ab7695b88f4f5c4ee8d297228241c6488b0e39b6d71cd2de9d0bb7a9790'
b'U\r\r\n'

(Pdb) cfile
'/tmp/pip-unpacked-wheel-lx3lxwld/setuptools/command/__pycache__/bdist_egg.cpython-38.pyc'

Within the same process the value returned by importlib._bootstrap_external._code_to_hash_pyc for the same source does not change:

> /usr/lib/python3.8/py_compile.py(169)compile()
-> bytecode = importlib._bootstrap_external._code_to_hash_pyc(
print(invalidation_mode) ; import hashlib ; hashlib.sha256(source_bytes).hexdigest() ; hashlib.sha256(source_hash).hexdigest() ; hashlib.sha256(importlib._bootstrap_external._code_to_hash_pyc(code, source_hash, (invalidation_mode == PycInvalidationMode.CHECKED_HASH))).hexdigest() ; importlib._bootstrap_external.MAGIC_NUMBER
PycInvalidationMode.CHECKED_HASH
'a33213258b106b1cedbec418662b29f4226a91e9f579dfc4218722f2a826a2b5'
'6750d9e9657c3ce3ad48843f9d615ece9310755cbfb9deb037c3c9ce2d5e249f'
'46827abbb669a51e7072be45d72353ee21db9b2b28f5a17bd6c957b0181baa64'
b'U\r\r\n'
(Pdb) print(invalidation_mode) ; import hashlib ; hashlib.sha256(source_bytes).hexdigest() ; hashlib.sha256(source_hash).hexdigest() ; hashlib.sha256(importlib._bootstrap_external._code_to_hash_pyc(code, source_hash, (invalidation_mode == PycInvalidationMode.CHECKED_HASH))).hexdigest() ; importlib._bootstrap_external.MAGIC_NUMBER
PycInvalidationMode.CHECKED_HASH
'a33213258b106b1cedbec418662b29f4226a91e9f579dfc4218722f2a826a2b5'
'6750d9e9657c3ce3ad48843f9d615ece9310755cbfb9deb037c3c9ce2d5e249f'
'46827abbb669a51e7072be45d72353ee21db9b2b28f5a17bd6c957b0181baa64'
b'U\r\r\n'

importlib._bootstrap_external._code_to_hash_pyc
https://github.com/python/cpython/blob/0f5a28f834bdac2da8a04597dc0fc5b71e50da9d/Lib/importlib/_bootstrap_external.py#L601-L609
def _code_to_hash_pyc(code, source_hash, checked=True):
    "Produce the data for a hash-based pyc."
    data = bytearray(MAGIC_NUMBER)
    flags = 0b1 | checked << 1
    data.extend(_pack_uint32(flags))
    assert len(source_hash) == 8
    data.extend(source_hash)
    data.extend(marshal.dumps(code))
    return data