python-cryptography missing cffi dependency

[Graham Hayes]

When installing keystone (via apt-get install keystone) apt does not install python-cffi, which is required for keystone to implement fernat tokens (the current recommended token type)

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: keystone 2:13.0.0-0ubuntu1~cloud0 [origin: Canonical]
Uname: Linux 4.4.88-mainline-rev1 x86_64
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Thu Mar 1 17:19:34 2018
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: keystone
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.keystone.keystone.conf: 2018-03-01T16:51:52.734036

[Corey Bryant (corey.bryant)]

Hi Graham,

Thanks for reporting this. It would be good if this was specified in upstream keystone's requirements as well, so adding upstream to the bug.

Thanks,
Corey

[Corey Bryant (corey.bryant)]

I'm not seeing any imports of ffi/cffi in keystone but maybe my grep is failing me.

[Corey Bryant (corey.bryant)]

Graham, sorry can you provide some more details? Is this possible an indirect dependency via another project?

[Graham Hayes (grahamhayes)]

The stack trace I am getting looks like:

    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
        result = method(req, **params)
      File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 154, in authenticate_for_token
        parent_audit_id=token_audit_id)
      File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 116, in wrapped
        __ret_val = __f(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 210, in issue_token
        parent_audit_id=parent_audit_id)
      File "/usr/lib/python2.7/dist-packages/keystone/token/providers/fernet/core.py", line 53, in issue_token
        *args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/keystone/token/providers/common.py", line 605, in issue_token
        token_id = self._get_token_id(token_data)
      File "/usr/lib/python2.7/dist-packages/keystone/token/providers/fernet/core.py", line 187, in _get_token_id
        app_cred_id=app_cred_id
      File "/usr/lib/python2.7/dist-packages/keystone/token/token_formatters.py", line 160, in create_token
        token = self.pack(serialized_payload)
      File "/usr/lib/python2.7/dist-packages/keystone/token/token_formatters.py", line 79, in pack
        return self.crypto.encrypt(payload).rstrip(b'=').decode('utf-8')
      File "/usr/lib/python2.7/dist-packages/cryptography/fernet.py", line 135, in encrypt
        return self._fernets[0].encrypt(msg)
      File "/usr/lib/python2.7/dist-packages/cryptography/fernet.py", line 51, in encrypt
        return self._encrypt_from_parts(data, current_time, iv)
      File "/usr/lib/python2.7/dist-packages/cryptography/fernet.py", line 62, in _encrypt_from_parts
        ciphertext = encryptor.update(padded_data) + encryptor.finalize()
      File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/ciphers/base.py", line 149, in update
        return self._ctx.update(data)
      File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 120, in update
        n = self.update_into(data, buf)
      File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 131, in update_into
        "unsigned char *", self._backend._ffi.from_buffer(buf)
    TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

It looks like it is transitive via cryptography when using fernat encryption.

[Corey Bryant (corey.bryant)]

Ok thanks, that's helpful. Something seems to be wrong with the python-cryptography package. It seems to intend to install cffi but it is obviously not happening.

[Corey Bryant (corey.bryant)]

I've opened the following bug against the Debian package: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892112

[Corey Bryant (corey.bryant)]

I think our long-term fix will come from the Debian maintainer's approach to adding dependencies. I'm going to fix this in Ubuntu for now by adding python-cffi to debian/control Depends.

[Corey Bryant (corey.bryant)]

I've uploaded a new fixed version of the python-cryptography to all affected releases. The Xenial (Mitaka) and Artful (Pike) versions will need to be reviewed by the SRU team before they are accepted into proposed and backported to the cloud-archive.

[Corey Bryant (corey.bryant)]

Graham, would you mind testing with a PPA? If you can let me know what release combination you're working with and I can get a PPA built.

I've unsubscribed the SRU team for now as Tristan mentioned in the Debian bug that he didn't immediately see what the stack trace in the Ubuntu bug report has to do with cffi being installed or not.

[Graham Hayes (grahamhayes)]

Sure, no problem - I can spin up a VM to test tomorrow.

I am running Xenial / Queens.

[Corey Bryant (corey.bryant)]

Graham, thanks that would be great. This will get you the new version:

sudo add-apt-repository ppa:ubuntu-cloud-archive/queens-staging
sudo apt update

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-cryptography - 2.1.4-1ubuntu1

---------------
python-cryptography (2.1.4-1ubuntu1) bionic; urgency=medium

  * d/control: Add python-cffi to binary package dependencies (LP: #1752660).

 -- Corey Bryant <email address hidden> Mon, 05 Mar 2018 14:37:41 -0500

[Corey Bryant (corey.bryant)]

Hi Graham,

I don't think this is a dependency issue. I'm going to back out the changes I made earlier as I pulled the trigger too quickly with those. What would be good to know is what 'buf' is set to in your scenario. Here's a little script that hits the same path your traceback goes down. It runs successfully on xenial-queens for me.

from cryptography.fernet import Fernet, MultiFernet
key1 = Fernet(Fernet.generate_key())
key2 = Fernet(Fernet.generate_key())
f = MultiFernet([key1, key2])
token = f.encrypt(b"Secret message!")

Thanks,
Corey

[Corey Bryant (corey.bryant)]

I've marked this incomplete until we can recreate it.

[Graham Hayes (grahamhayes)]

Yeah - I have been trying to repro on a new machine, and I can't.

It looks like it may have been a weird machine state due to it being an long running machine that had the archive added after ~ 6 months of running