include_system_packages is True and may obscure dependency isolation in venvs

Bug #1760956 reported by Ryan Beisner on 2018-04-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi Charm
Medium
Unassigned
OpenStack AODH Charm
Medium
Unassigned
OpenStack Barbican Charm
Medium
Unassigned
OpenStack Barbican SoftHSM Charm
Medium
Unassigned
OpenStack Ceph-FS Charm
Medium
Unassigned
OpenStack Designate Charm
Medium
Unassigned
OpenStack Designate-Bind Charm
Medium
Unassigned
OpenStack Keystone LDAP integration
Medium
Unassigned
OpenStack Manila Charm
Medium
Unassigned
OpenStack Manila Generic Backend Charm
Medium
Unassigned
OpenStack Tempest Charm
Medium
Unassigned
OpenStack neutron-api-odl charm
Medium
Unassigned
OpenStack openvswitch-odl charm
Medium
Unassigned
charms.openstack
Medium
Unassigned

Bug Description

Use of include_system_packages blurs the lines between libraries which are declared and built into a python virtualenv, and libraries which may or may not exist on the underlying distribution.

There is risk in that, and we should aim for full dependency isolation by setting this to False.

This should be coupled with an explicit declaration of use_venv: True, as use_venv has differing defaults across time and space.

Some (but not all) examples:

https://github.com/openstack/charm-aodh/blob/master/src/layer.yaml

https://github.com/openstack/charm-gnocchi/blob/master/src/layer.yaml

https://github.com/openstack/charm-designate/blob/master/src/layer.yaml

Still need to do a full grep to assess current use_venv and include_system_packages settings currently in the full suite reactive OpenStack charms.

Ryan Beisner (1chb1n) wrote :

The following confirms and details the current state of the two options, which are essentially at odds with one-another in terms of declaring dependencies in a sane way.

ie. When using a venv, we should never include system packages.

#### All src (reactive) OpenStack charms:
ubuntu@beisner-bastion:~/git/release-tools⟫ find -type f -name layer.yaml | sort
./aodh/src/layer.yaml
./barbican-softhsm/src/layer.yaml
./barbican/src/layer.yaml
./ceph-fs/src/layer.yaml
./designate-bind/src/layer.yaml
./designate/src/layer.yaml
./gnocchi/src/layer.yaml
./keystone-ldap/src/layer.yaml
./manila-generic/src/layer.yaml
./manila/src/layer.yaml
./neutron-api-odl/src/layer.yaml
./openvswitch-odl/src/layer.yaml
./tempest/src/layer.yaml

#### Those which take a stance on use_venv and/or include_system_packages:
ubuntu@beisner-bastion:~/git/release-tools⟫ find -type f -name layer.yaml | sort | xargs egrep "system|venv"
./aodh/src/layer.yaml: use_venv: True
./aodh/src/layer.yaml: include_system_packages: True
./barbican-softhsm/src/layer.yaml: use_venv: true
./barbican-softhsm/src/layer.yaml: include_system_packages: true
./barbican/src/layer.yaml: use_venv: true
./barbican/src/layer.yaml: include_system_packages: true
./designate/src/layer.yaml: use_venv: True
./designate/src/layer.yaml: include_system_packages: True
./gnocchi/src/layer.yaml: use_venv: True
./gnocchi/src/layer.yaml: include_system_packages: True
./keystone-ldap/src/layer.yaml: use_venv: True
./keystone-ldap/src/layer.yaml: include_system_packages: True
./manila/src/layer.yaml: use_venv: true
./manila/src/layer.yaml: include_system_packages: true
./neutron-api-odl/src/layer.yaml: use_venv: True
./neutron-api-odl/src/layer.yaml: include_system_packages: True
./openvswitch-odl/src/layer.yaml: use_venv: True
./openvswitch-odl/src/layer.yaml: include_system_packages: True

#### Those which do not declare either, leaving it up to version chance:
ceph-fs
designate-bind
manila-generic
tempest

Ryan Beisner (1chb1n) wrote :
summary: - include_system_packages is True and may obscure dependency isolation
+ include_system_packages is True and may obscure dependency isolation in
+ venvs
Ryan Beisner (1chb1n) wrote :

In an interesting and relevant twist, coincidentally the same day we're dealing with pip >= 10.0 and python-apt on a separate bug which affected test execution environments (bug #1760720):

At least in the tempest charm, if I leave use_venv True and set use_system_packages False, I get apt import errors at charm run time. The charms are clearly leaning on the fact that some system-installed libraries are installed outside of the virtualenv (via debs in this case). Lines, blurred. Wires, crossed.

This is just one example, but it is a clear example of how we have missed fully declaring dependencies from a pure python standpoint.

Further ideas, debate, conversation are of course welcome.

Frode Nordahl (fnordahl) on 2018-04-17
Changed in charm-gnocchi:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-aodh:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-barbican:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-barbican-softhsm:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-ceph-fs:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-designate:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-designate-bind:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-keystone-ldap:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-manila:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-manila-generic:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-tempest:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-neutron-api-odl:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-openvswitch-odl:
importance: Undecided → Medium
status: New → Triaged
Changed in charms.openstack:
importance: Undecided → Medium
status: New → Triaged
Liam Young (gnuoy) on 2019-08-27
Changed in charm-designate:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers