Regressions in 2.39.2 in xenial-updates in classic snaps (relative to 2.35)

[Nish Aravamudan]

git-ubuntu builds automatically on Launchpad and I happened to hit a window where one build passed and one did not build quite correctly. In between the snapcraft update hit ftpmaster.

I see:

Preparing to build distro-info
Building distro-info
You must give at least one requirement to install (maybe you meant "pip install /build/git-ubuntu/parts/distro-info/python-packages"?)

and

Determining the version from the project repo (version: git).
The version has been set to '0.7.1-dirty'
Snapping 'git-ubuntu' ...

Snapped git-ubuntu_0.7.1-dirty_amd64.snap

The former worked before the snapcraft update; the latter is clearly not true, as it's building via a Git recipe on Launchpad.

[Nish Aravamudan (nacc)]

Worked: https://launchpadlibrarian.net/358999394/buildlog_snap_ubuntu_xenial_amd64_git-ubuntu_BUILDING.txt.gz

Not working: https://launchpadlibrarian.net/359014459/buildlog_snap_ubuntu_xenial_amd64_git-ubuntu_BUILDING.txt.gz

[Sergio Schvezov (sergiusens)]

https://git.launchpad.net/usd-importer?h=master

[Sergio Schvezov (sergiusens)]

Potential PR that broke it https://github.com/snapcore/snapcraft/pull/1607/files#diff-07f6bf373bfe0a8212f1c9fb7050c835L352

Use of sourcedir to find setup.py instead of builddir

[Nish Aravamudan (nacc)]

Sigh, there's another regression it looks like (with sergiusens' ppa):

With 2.35:

$ snap run --shell git-ubuntu
wnacc@pitfall:~$ which gpgv
/snap/git-ubuntu/x21/usr/bin/gpgv
nacc@pitfall:~$ ldd `which gpgv`
 linux-vdso.so.1 => (0x00007ffc73fd2000)
 libz.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libz.so.1 (0x00007f6c95623000)
 libbz2.so.1.0 => /snap/core/current/lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f6c95413000)
 libgcrypt.so.20 => /snap/git-ubuntu/current/usr/lib/libgcrypt.so.20 (0x00007f6c950b2000)
 libgpg-error.so.0 => /snap/git-ubuntu/current/usr/lib/libgpg-error.so.0 (0x00007f6c94e99000)
 libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007f6c94acf000)
 /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007f6c9583d000)

with PPA snapcraft:

$ snap run --shell git-ubuntu
ubuntu@nacc-bastion:~$ which gpgv
/snap/git-ubuntu/378/usr/bin/gpgv
ubuntu@nacc-bastion:~$ ldd `which gpgv`
        linux-vdso.so.1 => (0x00007fff98bee000)
        libz.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libz.so.1 (0x00007f6d3d627000)
        libbz2.so.1.0 => /snap/core/current/lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f6d3d417000)
        libgcrypt.so.20 => /snap/core/current/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f6d3d136000)
        libgpg-error.so.0 => /snap/git-ubuntu/378/usr/bin/../../lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f6d3cf21000)
        libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007f6d3cb57000)
        /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007f6d3d841000)

This is a serious regression for classic snaps that were wroking before. And are fundamentally broken now.

[Sergio Schvezov (sergiusens)]

This is strange, this is what I see

ubuntu@snapcraft-bionic-dev:~$ ldd /snap/git-ubuntu/x1/usr/bin/gpgv
        linux-vdso.so.1 => (0x00007ffc1e2e5000)
        libz.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libz.so.1 (0x00007ff86b323000)
        libbz2.so.1.0 => /snap/core/current/lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007ff86b113000)
        libgcrypt.so.20 => /snap/git-ubuntu/x1/usr/bin/../lib/libgcrypt.so.20 (0x00007ff86adb0000)
        libgpg-error.so.0 => /snap/git-ubuntu/x1/usr/bin/../lib/libgpg-error.so.0 (0x00007ff86ab9a000)
        libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007ff86a7d0000)
        /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007ff86b53d000)
ubuntu@snapcraft-bionic-dev:~$ realpath /snap/git-ubuntu/x1/usr/bin/../lib/libgcrypt.so.20
/snap/git-ubuntu/x1/usr/lib/libgcrypt.so.20.2.1

[Sergio Schvezov (sergiusens)]

Can I see the logs? We now parse and generate $ORIGIN during prime for whatever is in the snap instead of blindly setting a subset of common library paths

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/1977

[Sergio Schvezov (sergiusens)]

Ok, I will be stuck on an airplane all day today, so please consider a revert if no one from the team can follow-up

[Sergio Schvezov (sergiusens)]

Just consider the revert, this is already a critical issue and has proven not to be a quick fix as would have thought .

[Sergio Schvezov (sergiusens)]

Regardless of the revert call, I think I found the root of all problems and added it to the above PR and to the PPA Nish is using to test.

[Nish Aravamudan (nacc)]

Kicking off a git-ubuntu rebuild now using the updated PPA version. Will report back in ~30 minutes.

I'll leave the revert decision up to the SRU team.

[Nish Aravamudan (nacc)]

Hrm, it doesn't seem to have helped at all:

ubuntu@nacc-bastion:~$ ldd `which gpgv`
        linux-vdso.so.1 => (0x00007ffca9557000)
        libz.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libz.so.1 (0x00007fa53d8d8000)
        libbz2.so.1.0 => /snap/core/current/lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fa53d6c8000)
        libgcrypt.so.20 => /snap/core/current/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fa53d3e7000)
        libgpg-error.so.0 => /snap/core/current/lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fa53d1d3000)
        libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007fa53ce09000)
        /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007fa53daf2000)

Now nothing is coming from the snap's library :(

[Kyle Fazzari (kyrofa)]

Try ppa:kyrofa/1752481. With that I get:

$ ldd /snap/git-ubuntu/x1/usr/bin/gpgv
 linux-vdso.so.1 => (0x00007ffd40d56000)
 libz.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libz.so.1 (0x00007fcb20e93000)
 libbz2.so.1.0 => /snap/core/current/lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fcb20c83000)
 libgcrypt.so.20 => /snap/git-ubuntu/x1/usr/bin/../lib/libgcrypt.so.20 (0x00007fcb20920000)
 libgpg-error.so.0 => /snap/git-ubuntu/x1/usr/bin/../lib/libgpg-error.so.0 (0x00007fcb20707000)
 libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007fcb2033d000)
 /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007fcb210ad000)

However, the selftest still fails because xz is looking on the system for lzma for some reason:

$ ldd $(which xz)
/snap/git-ubuntu/x1/usr/bin/xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by /snap/git-ubuntu/x1/usr/bin/xz)
 linux-vdso.so.1 => (0x00007ffc17177000)
 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc7ffc2e000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc7ffa11000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc7ff647000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc7ff443000)
 /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007fc7ffe50000)

[Kyle Fazzari (kyrofa)]

Note that I've noticed the same issue with mksquashfs in snapcraft itself, LP: #1752964.

[Kyle Fazzari (kyrofa)]

Okay check this out. I removes all parts from the project except dpkg and xz-utils to speed things up.

If I have a clean tree and run `snapcraft prime` this is what I end up with:

$ ldd prime/usr/bin/xz
prime/usr/bin/xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by prime/usr/bin/xz)
 linux-vdso.so.1 => (0x00007ffcfebf4000)
 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fbb73c45000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbb73a28000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbb7365e000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbb7345a000)
 /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007fbb73e67000)

Bad. If I clean the tree and now run `snapcraft prime dpkg` followed by `snapcraft prime xz-utils` I get this:

$ ldd prime/usr/bin/xz
 linux-vdso.so.1 => (0x00007ffc261ea000)
 liblzma.so.5 => /home/ubuntu/usd-importer/prime/usr/bin/../lib/liblzma.so.5 (0x00007fae8bbdb000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fae8b9be000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fae8b5f4000)
 /snap/core/current/lib64/../lib/x86_64-linux-gnu/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007fae8be11000)

Good! I suspect this is Snapcraft not properly re-evaluating its environment as it executes its lifecycle.

[Kyle Fazzari (kyrofa)]

Or perhaps it's not properly setting rpaths for libraries coming from the same part unless those directories already exist.

[Kyle Fazzari (kyrofa)]

Got it. Hello global state!

The problem is that dpkg is a prerequisite of xz-utils, but _also_ requires liblzma. As a result, when priming them one after the other, we cache the lookup result for dpkg, and re-use it when priming xz-utils:

    https://github.com/snapcore/snapcraft/blob/master/snapcraft/internal/elf.py#L118

When running them one after the other, the cache is cleared, and the libs are properly found.

[Kyle Fazzari (kyrofa)]

We probably need the cache to be either per-part, or do the rpath patching after all parts have been primed.

[Kyle Fazzari (kyrofa)]

(note that we already reset the cache for each part, but only for libs that weren't found)

[Kyle Fazzari (kyrofa)]

Fix for that last bit about the cache: https://github.com/snapcore/snapcraft/pull/1980

[Steve Langasek (vorlon)]

Hello Nish, or anyone else affected,

Accepted snapcraft into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapcraft/2.39.3+really2.35 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Sergio Schvezov (sergiusens)]

Thanks for following up Kyle.

[Kyle Fazzari (kyrofa)]

With those last few PRs applied:

$ git-ubuntu.self-test
pylint passed!
========================================= test session starts =========================================
platform linux -- Python 3.6.3, pytest-3.1.3, py-1.5.2, pluggy-0.4.0
rootdir: /snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu, inifile:
plugins: cov-2.5.1
collected 165 items

../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/apt_repo_test.py .......................
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/build.py ........
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/git_repository.py ........
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/lint.py ..........
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/repo_builder.py ..............
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/scriptutils_test.py ......
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/source_builder_test.py ............
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/test_git_repository.py ...........................................
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/test_importer.py ...
../../../snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/versioning.py ......................................

----------- coverage: platform linux, python 3.6.3-final-0 -----------
Name Stmts Miss Cover
------------------------------------------------------------------------------------------------------
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/__init__.py 0 0 100%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/__main__.py 86 76 12%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/apt_repo.py 97 1 99%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/apt_repo_test.py 74 0 100%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/build.py 451 376 17%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/buildsource.py 36 25 31%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/cache.py 1 0 100%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/clone.py 75 56 25%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/dsc.py 65 45 31%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/exportorig.py 53 40 25%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/git_repository.py 1041 613 41%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/importer.py 572 492 14%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/importlocal.py 152 133 12%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/importppa.py 100 80 20%
/snap/git-ubuntu/x1/lib/python3.6/site-packages/gitubuntu/lint.py 366 277 24%
/snap/git-ubu...

[Alan Pope 🍺🐧🐱 🦄 (popey)]

I tested the version in proposed in a clean ubuntu 16.04 machine. It successfully created a working snap (git-ubuntu) which previously did not build (with 2.39x). I installed the proposed version of snapcraft over the top of the 2.39x version. Worked fine.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapcraft - 2.39.3+really2.35

---------------
snapcraft (2.39.3+really2.35) xenial; urgency=medium

  * Revert back to 2.35 due to SRU regression for classic snaps.
    - LP: #1752481

 -- Nishanth Aravamudan <email address hidden> Fri, 02 Mar 2018 10:10:00 -0800

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for snapcraft has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Sergio Schvezov (sergiusens)]

2.40 should have the required fixes