FFe: transitioning libraries to multiarch paths

[Steve Langasek]

(Filed without a package as there are a large number of packages affected)

With the next upload of dpkg, all the pieces will be in place to begin migrating our library packages to use the multiarch library paths for <https://blueprints.launchpad.net/ubuntu/+spec/packageselection-foundations-n-multiarch-support>. Individually the changes are low risk, the design has been agreed for over a year and I have aggressively tested in my ppa, but given the large number of affected packages and that this is ultimately a new feature rather than a bugfix, I think a FFe request is in order.

The overall plan is:
 - push a new dpkg version that implements support for the final library paths
 - bootstrap gcc and eglibc to use the paths (requires two gcc uploads with an intervening eglibc upload)
 - push debhelper and pkg-config patches to support multiarch
 - write up documentation for how to transition libraries, and announce this on ubuntu-devel
 - convert the desktop library stack, one library at a time

The transition documentation is not written yet, because it partially depends on the finer details of the patch accepted by debhelper upstream.

My target for this cycle, which I believe is a realistic one, is to get flashplugin-installer:i386 co-installable with an amd64 desktop stack. This involves converting just over 100 library packages to the multiarch paths (highly parallelizable once we get going). Is this acceptable to the release team, and when do we need to be done with these changes for natty? before beta?

Risks I'm aware of with this transition:
 - there are a handful of i386 packages in the archive that will be broken by the transition, because they install to the provisional library paths rather than the final ones we're using. These packages are lib32ffi-dev, libffi-dev, libhwloc-dev, liblouis-dev and liblouisxml-dev; they'll need to be fixed soon after we upload the new gcc.
 - any library that loads architecture-dependent plugins / modules will need additional care to ensure that the system is never left in a broken halfway-transitioned state. For example, I have a patch to pam to prepend a multiarch module path to the standard /lib/security path; whereas for glib I only have it patched to look in the new gio modules path but *not* the old one, resulting in a long list of Breaks: in my ppa package that need to be sorted before release (preferably the same way pam solves it).
 - ia32-libs will be broken in various ways by this change, because the libraries it rebundles will all come installed in the new multiarch paths and ia32-libs will need to adapt. (Debian policy 9.1.1 does not permit ia32-libs to install its contents to the multiarch paths; ia32-libs needs to be fixed up to make sure it only installs to /usr/lib32 instead to avoid file conflicts.) But even after this fix-up, there may be some regressions in functionality due to the changes in module paths mentioned in the previous point; this appears to affect glib and gtk in particular, for which ia32-libs ships symlinks under /usr/lib.
 - this is very much a one-way transition. Once we start down this path in earnest, it will be infeasible to roll it back for natty.
 - there has been no significant testing of multiarch enablement with package manager frontends such as synaptic, software-center, and aptitude, so it may not actually be feasible for users to turn on multiarch on the desktop for natty if they're using one of these frontends.

Do I have the release team's blessing to proceed?

[Scott Kitterman (kitterman)]

When will this dpkg change go into Debian? If we go first is there some risk
that the 'final' library locations won't be quite final after all?

[Kate Stewart (kate.stewart)]

Steve, thanks for flagging and the analysis.

Main reservation at this point is that we're already behind where we were in the maverick time frame with open bugs high/critical bugs for this point in the release. I'm a bit worried on the surprise "breakages", esp as we won't be able to roll back. Would like cjwatson and pitti to weigh in on this one.

That being said, if we do it, I'd rather it be uploaded on monday ( assuming cjwatson and pitti are ok with it), and IF we have confidence the transition can be finished (all libs fixed up and working again) by about 7 days before beta.
Otherwise we should wait for the start of Oneiric.

[Steve Langasek (vorlon)]

On Fri, Mar 11, 2011 at 09:18:29PM -0000, Scott Kitterman wrote:
> When will this dpkg change go into Debian? If we go first is there some risk
> that the 'final' library locations won't be quite final after all?

The library paths have been agreed with the Debian dpkg, eglibc, and gcc
maintainers; there should be no risk of change to the paths.

The one remaining point that's being discussed in dpkg upstream is the
precise form the dpkg on-disk database will take. We currently have a
preliminary implementation in natty that can be safely migrated from, but if
there are further changes to the database format those will cause
significant migration pain if we switch too early. So I'm currently waiting
for the current on-disk database to be signed off on upstream; I'm giving
this until Tuesday (Mar 15) to work its way through.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Steve Langasek (vorlon)]

> IF we have confidence the transition can be finished (all libs fixed up
> and working again) by about 7 days before beta.

I have no concerns about being able to get the archive into a usable state
for beta freeze wrt multiarch. We may not make it all the way up the stack
to flashplugin-installer in that timeframe, but this is a very soft
transition and flashplugin-installer is a very soft target. The only real
point of no return is that, once begun, we will wind up with *some*
libraries in a multiarch layout for natty.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Martin Pitt (pitti)]

Thanks Steve for the detailled plan.

"dpkg that implements support for the final library paths", "gcc and eglibc to use the paths", and "debhelper and pkg-config patches to support multiarch" will only enable additional library search paths, and thus our current libary packages should continue to work unmodified, right? That is, with the exception of lib32ffi-dev and friends; do you know (by grepping Contents.gz or similar) that this is the complete list? The steps until here (and the transition document, of course) seem low risk to me. As you said, there might be a potential dpkg db change, but we only need to fix that for intra-natty upgraders, and it shouldn't be too difficult. I agree with Kate that I'd like to have these changes in well before beta, so that we can do the major testing rounds with the new infrastructure in place.

I don't think we are in a hurry to update the library stack by beta. We should,, upload a small set of central libraries (i. e. if something breaks, we _will_ notice), such as glib2.0, libx11, libxcb, libpcre, and libdl (all of which are required by libflashplayer.so). gtk isn't needed by flashplayer, but converting that early would be interesting because it would test the robustness with runtime loaded platform specific modules (/usr/lib/gtk-2.0/2.10.0/immodules/*.so and /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders/*.so) We should then leave this in the archive for a few days, iron out the problems, and could then continue until beta-2.

Question here: Once the new debhelper is in, will library packages will be built for multiarch compatible paths by default, or does this need to be enabled in debian/rules?

Getting up to the point of flashplugin is a very laudable goal, as this is probably the #1 reason why people have ia32-libs installed. The flash player has tons of vulnerabilities, and so far people are using it with outdated libraries (which ia32-libs constantly is). So I agree that for this benefit it is worth taking some risk here.

[Steve Langasek (vorlon)]

On Mon, Mar 14, 2011 at 08:00:02AM -0000, Martin Pitt wrote:
> "dpkg that implements support for the final library paths", "gcc and
> eglibc to use the paths", and "debhelper and pkg-config patches to
> support multiarch" will only enable additional library search paths, and
> thus our current libary packages should continue to work unmodified,
> right?

Correct.

> That is, with the exception of lib32ffi-dev and friends; do you know (by
> grepping Contents.gz or similar) that this is the complete list?

Yes, this is a grep for 'usr/(lib|include)/i686-linux-gnu' in i386 and amd64
Contents.gz.

> I don't think we are in a hurry to update the library stack by beta. We
> should,, upload a small set of central libraries (i. e. if something
> breaks, we _will_ notice), such as glib2.0, libx11, libxcb, libpcre, and
> libdl (all of which are required by libflashplayer.so).

I have a pretty good set of libraries bootstrapped at
https://launchpad.net/~vorlon/+archive/multiarch/, FWIW. It's actually
enough to make flashplugin-installer cross-installable, but not
co-installable with a desktop yet.

> Question here: Once the new debhelper is in, will library packages will
> be built for multiarch compatible paths by default, or does this need to
> be enabled in debian/rules?

If using dh(1) and your package uses autoconf, you can switch to compat
level 9 and the package will build for multiarch paths by default. It's not
safe to make this a completely automatic conversion:

 - a new pre-depends line needs to be added to the library to ensure that
   the virtual package 'multiarch-support' (provided by a compatible libc)
   is installed before we unpack libraries to the new path.
 - various embedded paths in debian/*.install, debian/*.links, etc will be
   invalidated, and not all of these will be detected at build time
 - patches may be needed for backwards-compatible plugin directory handling

So I'll make this as easy as possible to switch, but it's not automatic.
And I see no good way to simplify the conversion for cdbs, which doesn't use
dh_auto_configure. I guess cdbs could itself key on debian/compat >= 9 and
pass the --libdir option automatically, but as this behavior change would not
be nearly as discoverable with cdbs as it is with debhelper, I prefer not to
get too clever.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Martin Pitt (pitti)]

Steve Langasek [2011-03-14 17:55 -0000]:
> If using dh(1) and your package uses autoconf, you can switch to compat
> level 9 and the package will build for multiarch paths by default. It's not
> safe to make this a completely automatic conversion:

Ah, thanks for the details. I was actually hoping that it wouldn't be
automatic :-) Using the dh compat level sounds like a very elegant
solution for this.

Thanks, looking forward to seeing this!

Martin

[Dmitry Tantsur (divius)]

Hell, you're mad if you give a FFE for such a HUGE change. Count broken things: Python build for source is the first (bug 738213). What a childish decision...

[Dmitry Tantsur (divius)]

> build FROM source
obvious fix.

[Steve Langasek (vorlon)]

Dmitry,

Thank you for bringing bug #738213 to our attention. I am committed to resolving those issues introduced by the upload of multiarch, and I believe the release team understands that, which is why the freeze exception has been granted.

I'm having a look at the python2.7 build and we will get the source package build fixed in time for beta. While you may think breaking python is proof that the transition as a whole is broken, there are a finite number of build systems that implement their own library resolution and need to be fixed. We've fixed several others already for this transition; python is another and will be taken care of ASAP.

A full rebuild of the Ubuntu main archive will also be scheduled to identify any other unexpected breakage introduced by this change. I believe the risk to the 11.04 release overall is still low.

[Gytis Raciukaitis (noxxious)]

Hi,
the possible breakage's I see now is for people using thirdparty i386 apps force-arch installed on 64bit systems and then the deps resolved using ia32-libs or using the famous getlibs script. The applications to be listed are lotus-notes, symphony, sametime, adobeair (these are the ones I'm forced to use). With the current state of multiarch in dpkg it's now not possible to get them installed due to dependencies and dependencies in i386 are not yet built for multiarch.

[Steve Langasek (vorlon)]

forcearch installation will work the same as before (i.e., not very well). ia32-libs will continue to work.

I don't know what "getlibs" is, so I disagree with your assertion that it's famous. But if it breaks, then it wasn't very robust to begin with. I guess the authors of this tool will need to update it for multiarch compatibility.

[Steve Langasek (vorlon)]

All the libraries that are going to be uploaded for multiarch to natty have been uploaded. Any further library changes will be staged in my ppa for this cycle, pushing to Debian ASAP and oneiric when it opens.

There's still some cleanup to be done of the fallout here (30ish packages in main FTBFS, a significant number of these are due to multiarch incompatibilities; 140ish libraries in universe need no-change rebuilds due to moved .la files; other packages will probably ftbfs in universe but no list available yet). We'll work through these as quickly as possible. If the release team knows of any other problems caused by the multiarch transition, please shout.