Python 3.10 in core22-base snap on bionic fails importing ssl module

[Sergio Rabellino]

I have a maas installed/running on an Ubuntu 18.04 host with snap 3.2.6-12016-g.19812b4da.
I would like to try the 3.3 beta and I did a "snap refresh --channel=3.3/beta maas".

The output follows:

2022-11-09T16:43:44Z INFO Waiting for "snap.maas.supervisor.service" to stop.
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Traceback (most recent call last):
  File "/snap/maas/x1/bin/maas", line 5, in <module>
    from maascli import main
  File "/snap/maas/x1/lib/python3.10/site-packages/maascli/__init__.py", line 10, in <module>
    from maascli.parser import get_deepest_subparser, prepare_parser
  File "/snap/maas/x1/lib/python3.10/site-packages/maascli/parser.py", line 11, in <module>
    from maascli import api
  File "/snap/maas/x1/lib/python3.10/site-packages/maascli/api.py", line 19, in <module>
    import httplib2
  File "/snap/maas/x1/usr/lib/python3/dist-packages/httplib2/__init__.py", line 40, in <module>
    import ssl
  File "/usr/lib/python3.10/ssl.py", line 98, in <module>
    import _ssl # if we can't import it, let the error propagate
ImportError: /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so: failed to map segment from shared obj
ect
-----)

I was not able to understand why it's happening. the _ssl.cpython-310-x86_64-linux-gnu.so library seems missing either from my system and from the snap (I unsquashed maas_24685.snap to check the file existance, but none found).
As a test I also tried a snap try maas_24685.snap, but the error is the same.

Note that installing a fresh/unconfigured maas 3.2.6 and then refreshing to 3.3 it does not happen...

[Anton Troyanov (troyanov)]

I can confirm that update doesn't work on bionic, but it worked with focal and jammy.

MAAS installed from 3.2/stable inside LXC container running ubuntu:bionic

root@cosmic-crayfish:~# snap install maas
Download snap "maas" (23947) from channel "3.2/stable"
maas (3.2/stable) 3.2.6-12016-g.19812b4da from Canonical✓ installed

root@cosmic-crayfish:~# snap refresh --channel=3.3/beta maas
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Traceback (most recent call last):
  File "/snap/maas/24685/bin/maas", line 5, in <module>
    from maascli import main
  File "/snap/maas/24685/lib/python3.10/site-packages/maascli/__init__.py", line 10, in <module>
    from maascli.parser import get_deepest_subparser, prepare_parser
  File "/snap/maas/24685/lib/python3.10/site-packages/maascli/parser.py", line 11, in <module>
    from maascli import api
  File "/snap/maas/24685/lib/python3.10/site-packages/maascli/api.py", line 19, in <module>
    import httplib2
  File "/snap/maas/24685/usr/lib/python3/dist-packages/httplib2/__init__.py", line 40, in <module>
    import ssl
  File "/usr/lib/python3.10/ssl.py", line 98, in <module>
    import _ssl # if we can't import it, let the error propagate
ImportError: /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so: failed to map segment from shared object
-----)

[Alberto Donato (ack)]

This happens even with a clean install of 3.3/beta on a bionic install:

root@b:~# snap install maas --channel=3.3/beta
2022-11-10T09:07:08Z INFO Waiting for automatic snapd restart...
error: cannot perform the following tasks:
- Run install hook of "maas" snap if present (run hook "install":
-----
Traceback (most recent call last):
  File "/snap/maas/24685/bin/maas", line 5, in <module>
    from maascli import main
  File "/snap/maas/24685/lib/python3.10/site-packages/maascli/__init__.py", line 10, in <module>
    from maascli.parser import get_deepest_subparser, prepare_parser
  File "/snap/maas/24685/lib/python3.10/site-packages/maascli/parser.py", line 11, in <module>
    from maascli import api
  File "/snap/maas/24685/lib/python3.10/site-packages/maascli/api.py", line 19, in <module>
    import httplib2
  File "/snap/maas/24685/usr/lib/python3/dist-packages/httplib2/__init__.py", line 40, in <module>
    import ssl
  File "/usr/lib/python3.10/ssl.py", line 98, in <module>
    import _ssl # if we can't import it, let the error propagate
ImportError: /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so: failed to map segment from shared object
-----)

[Jerzy Husakowski (jhusakowski)]

The current assumption is that there's an issue in snapcraft/snapd (core22 on bionic with python hooks). Please see if this can be reproduced.

[Alberto Donato (ack)]

So, this seems to be a snapd/core22 issue.

The issue can be seen by installing a core22-based snap on bionic, then getting a shell with snap run --shell and running `python -c 'import ssl'`

It can be reproduced with a very simple snap like:

--- snapcraft.yaml
name: test-22-on-bionic
version: '1.0'
summary: test snap
description: test snap
confinement: strict
grade: stable
base: core22

apps:
  test-22-on-bionic:
    command: script

parts:
  local:
    plugin: dump
    source: .
    stage:
      - script

--- script
#!/usr/bin/env python3

import ssl

installing the resulting snap on bionic shows the error:

root@b:~# test-22-on-bionic
Traceback (most recent call last):
  File "/snap/test-22-on-bionic/x2/script", line 3, in <module>
    import ssl
  File "/usr/lib/python3.10/ssl.py", line 98, in <module>
    import _ssl # if we can't import it, let the error propagate
ImportError: /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so: failed to map segment from shared object

[Michael Vogt (mvo)]

Thanks, I can reproduce this on 18.04, doing::
$ sudo snap install test-snapd-sh-core22
$ test-snapd-sh-core22.sh -c 'python3 -c "import ssl"'
is enough to trigger it. Fwiw, this works fine on a 22.04 host.

However on 18.04 I see a bunch of apparmor="DENIED" messages for operation="file_mmap". And indeed running it in devmode works. So it looks like our confinement needs updating (but it's bit unclear why only on 18.04 but not on 22.04) :/

[Michael Vogt (mvo)]

I pushed a fix in https://github.com/snapcore/snapd/pull/12308

[Adam Collard (adam-collard)]

With no change in MAAS, this has been fixed with the PR that @mvo pushed and landed in snapd 2.58

[Seth Tanner (sjtanner)]

It looks like this may still be a problem in 18.04
Currently running snapd 2.58+18.04

when attempting to upgrade from maas 3.2.6 to maas 3.3

sudo snap refresh --channel=3.3 maas
2023-02-09T22:34:01Z INFO Waiting for "snap.maas.supervisor.service" to stop.
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Traceback (most recent call last):
  File "/snap/maas/25850/bin/maas", line 5, in <module>
    from maascli import main
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/__init__.py", line 10, in <module>
    from maascli.parser import get_deepest_subparser, prepare_parser
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/parser.py", line 11, in <module>
    from maascli import api
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/api.py", line 27, in <module>
    from maascli import utils
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/utils.py", line 15, in <module>
    from OpenSSL import crypto
  File "/snap/maas/25850/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/snap/maas/25850/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 17, in <module>
    from OpenSSL._util import (
  File "/snap/maas/25850/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/snap/maas/25850/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 189, in <module>
    Binding.init_static_locks()
  File "/snap/maas/25850/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 163, in init_static_locks
    cls._ensure_ffi_initialized()
  File "/snap/maas/25850/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 151, in _ensure_ffi_initialized
    _openssl_assert(
  File "/snap/maas/25850/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 77, in _openssl_assert
    raise InternalError(
cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=126615813, lib=15, reason=786693, reason_text=b'error:078C0105:common libcrypto routines::init fail')])
-----)

[Sergio Rabellino (rabser)]

Tested just now on my MAAS 3.2.6-12016-g.19812b4da running on a Ubuntu 18.04.6: same error

root@playground:~# snap refresh --channel=3.3 maas
2023-02-10T08:22:13Z INFO Waiting for "snap.maas.supervisor.service" to stop.
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Traceback (most recent call last):
  File "/snap/maas/25850/bin/maas", line 5, in <module>
    from maascli import main
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/__init__.py", line 10, in <module>
    from maascli.parser import get_deepest_subparser, prepare_parser
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/parser.py", line 11, in <module>
    from maascli import api
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/api.py", line 27, in <module>
    from maascli import utils
  File "/snap/maas/25850/lib/python3.10/site-packages/maascli/utils.py", line 15, in <module>
    from OpenSSL import crypto
  File "/snap/maas/25850/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
... omissis as it's the same error of the preceding comment.