package-install-v2 breaks older distros

Bug #1415240 reported by Abel Lopez on 2015-01-27
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
diskimage-builder
High
Pino Toscano

Bug Description

package-installs-v2 uses subprocess.check_output which is new in python 2.7
Building a rhel6 image will fail as it ships with python 2.6
Traceback (most recent call last):
  File "/usr/local/bin/package-installs-v2", line 88, in <module>
    main()
  File "/usr/local/bin/package-installs-v2", line 57, in main
    map_output = subprocess.check_output(
AttributeError: 'module' object has no attribute 'check_output'

John Dewey (retr0h) on 2015-01-27
Changed in diskimage-builder:
assignee: nobody → John Dewey (retr0h)
Ben Nemec (bnemec) wrote :

This probably requires some discussion. OpenStack in general has dropped py26 support in Kilo, but diskimage-builder is a somewhat different situation because it targets a wider variety of distros.

Abel Lopez (al592b) wrote :

John was suggesting that we can do something along the lines of
https://review.openstack.org/#/c/10331/1/tools/hacking.py

Abel Lopez (al592b) wrote :

A workaround exists, which involves specifying package installs in the deprecated fashion. By utilizing the older file format (e.g. install.d/package-installs-${ELEMENT} ), the build succeeds.
This is labeled as deprecated in package-installs README, in favor of the yaml file (elements/${ELEMENT}/package-installs.yaml )

This means the older format must remain deprecated, but available, until py26 based distros are no longer needed.

Excerpts from Abel Lopez's message of 2015-01-29 22:15:23 UTC:
> A workaround exists, which involves specifying package installs in the deprecated fashion. By utilizing the older file format (e.g. install.d/package-installs-${ELEMENT} ), the build succeeds.
> This is labeled as deprecated in package-installs README, in favor of the yaml file (elements/${ELEMENT}/package-installs.yaml )
>
> This means the older format must remain deprecated, but available, until
> py26 based distros are no longer needed.
>

I'm inclined to say that we should probably define some min/max versions
for each distro that we build and state those in the README of the distro
element. Then, ideally we could make some basic CI jobs that build and
test those versions and exercises package installs, source-repositories,
etc.

As of now the closest thing we have to that is the tripleo-CI which
tests Ubuntu 14.04 and Fedora 20.

Abel Lopez (al592b) wrote :

Sounds reasonable. I wouldn't expect Ubuntu 10.04 or CentOS-5 to still work, especially given the old version of cloud-init they use. Saying something like "Ubuntu-12.04+ and EL-6+" isn't reaching too far

Gregory Haynes (greghaynes) wrote :

I think the py26 test not catching this comes down to us not really having a good unit test story for elements.

This is workaroundable but its not totally straightforwrard. You basically have to use subprocess.popen and poll for the return code. Id be up for doing this but I would really like to see us defining supported distro versions (as mentioned) and have some kind of way to CI this...

Abel Lopez (al592b) wrote :

What do you guys think about this:
http://paste.openstack.org/show/180854/

This gets legacy OS's built without errors (pending that my other reviews get merged)

Abel Lopez (al592b) wrote :

obviously, this means that elements that only supply packages as "install-packages.json" won't install on 2.6 python, but this allows us to provide a "Best Effort" for rhel/centos 6.

Luigi Toscano (ltoscano) wrote :

CentOS/RHEL6 is definitely needed. Basically all Sahara images for CentOS/RHEL6 support only version 6 at this time. Sahara relies on diskimage-builder and shara-image-elements to build images.

Abel Lopez (al592b) wrote :

CentOS6 can be built, but it requires major modifications to redhat-common element (specifically the tar sections, no support for --xattr-include/exclude), an update to pkg-map to map centos to redhat family, and the above mentioned update to package-installs.

Fix proposed to branch: master
Review: https://review.openstack.org/159050

Changed in diskimage-builder:
assignee: John Dewey (retr0h) → Pino Toscano (ptoscano)
status: New → In Progress
Changed in diskimage-builder:
importance: Undecided → High

Reviewed: https://review.openstack.org/159050
Committed: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=7f410aaff211a1d84f63e7ddf63b0a8badebea0a
Submitter: Jenkins
Branch: master

commit 7f410aaff211a1d84f63e7ddf63b0a8badebea0a
Author: Pino Toscano <email address hidden>
Date: Wed Feb 25 17:46:08 2015 +0100

    package-installs: work with Python < 2.7

    subprocess.check_output() has been introduced in Python 2.7, so the
    script will fail when trying to install stuff in guests with Python 2.6
    and older (like RHEL 6 / CentOS 6, for example).

    Thus gracefully fallback to subprocess.Popen() when
    subprocess.check_output() is not available.

    Change-Id: I335148397932177810f095a942b993b249991107
    Closes-Bug: #1415240

Changed in diskimage-builder:
status: In Progress → Fix Committed
Abel Lopez (al592b) wrote :

Looks good

Changed in diskimage-builder:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers