[SRU] intltool confused by separate build-dir

Bug #1117944 reported by Zbigniew Jędrzejewski-Szmek
66
This bug affects 10 people
Affects Status Importance Assigned to Milestone
intltool
Confirmed
Undecided
Unassigned
intltool (Ubuntu)
Fix Released
High
Unassigned
Xenial
Fix Released
Undecided
Gianfranco Costamagna
Artful
Fix Released
Undecided
Gianfranco Costamagna

Bug Description

[Impact]

An old bug in intltool breaks out-of-tree builds, including "make distcheck", in a lot of intltool-using projects when automake version is 1.15 or newer.

The fix for this issue (by Aleksander Morgado) is provided in the debdiffs attached to the report.

It's a long-standing bug which is present in all current Ubuntu releases,
starting from Xenial. Would be really nice to get it finally fixed. The developers have to patch their intltool locally just to have a working distcheck target (which is used when making release tarballs).

[Test Case]

Try making release tarball for some upstream package. For example, it can be mate-desktop, a base project for various MATE components.

$ sudo apt-get build-dep mate-desktop
$ git clone https://github.com/mate-desktop/mate-desktop
$ cd mate-desktop
$ ./autogen.sh --enable-gtk-doc --enable-deprecated --disable-strict
$ make -j5 && make dist -j5 && make distcheck -j5

This will build the project, make release tarball and check it by unpacking it into a new directory and building (again) in it. The last step will fail:

-----------------------------------

srcdir=../../../po /usr/bin/intltool-update -m
The following files contain translations and are currently not in use. Please
consider adding these to the POTFILES.in file, located in the po/ directory.

sub/mate-about/mate-about.desktop.in
sub/tools/mate-color-select.desktop.in

If some of these files are left out on purpose then please add them to
POTFILES.skip instead of POTFILES.in. A file 'missing' containing this list
of left out files has been written in the current directory.
Please report to https://github.com/mate-desktop/mate-desktop/
if [ -r missing -o -r notexist ]; then \
  exit 1; \
fi
Makefile:179: recipe for target 'check' failed

-----------------------------------

The out-of-tree build breaks here.

[Regression Potential]

None. This is a fix for a broken feature, it doesn't affects other ones.
Also the fix had been tested by MATE developers in various distros, including Debian, Ubuntu and Fedora.

[Original Description]

systemd uses 'intltool-update -m' from intltoolize in its 'make check'.
$(top_srcdir)/po/POTFILES.skip contains the name of a generated file (src/core/org.freedesktop.systemd1.policy.in), which will be generated as $(top_builddir)/src/core/org.freedesktop.systemd1.policy.in. If $(builddir)==$(srcdir) than everything works fine. When they are different, there's no way to instruct intltool-update to skip the file. Since the name of the build directory is not known and can be arbitrary, there should be a way to instruct intltool-update to ignore files relative to the $(top_builddir), or something like that. One option would be to turn the file list into a list of globs (**/src/core/org.freedesktop.systemd1.policy.in would work perfectly in our case).

Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

Reproduced with intltool bzr master.

This issue gains importance due to the following change in automake 1.15:

The make distcheck target has been changed to use $(distdir)/_build/sub as its working directory. Code making assumption on the value of srcdir and top_srcdir during make distcheck will fail.

When testing with automake 1.15, you may end up seeing that the files generated during compilation into builddir are not ignored from intltool-update -m even if they are listed in POTFILES.skip, e.g.:

make[2]: Entering directory '/home/aleksander/Development/foss/ModemManager/ModemManager-1.5.0/_build/sub/po'
INTLTOOL_EXTRACT="/usr/bin/intltool-extract" XGETTEXT="/usr/bin/xgettext" srcdir=../../../po /usr/bin/intltool-update --gettext-package ModemManager --pot
rm -f missing notexist
srcdir=../../../po /usr/bin/intltool-update -m
The following files contain translations and are currently not in use. Please
consider adding these to the POTFILES.in file, located in the po/ directory.

sub/data/org.freedesktop.ModemManager1.policy.in

If some of these files are left out on purpose then please add them to
POTFILES.skip instead of POTFILES.in. A file 'missing' containing this list
of left out files has been written in the current directory.
Please report to <email address hidden>
if [ -r missing -o -r notexist ]; then \
  exit 1; \
fi
Makefile:150: recipe for target 'check' failed
make[2]: *** [check] Error 1
make[2]: Leaving directory '/home/aleksander/Development/foss/ModemManager/ModemManager-1.5.0/_build/sub/po'
Makefile:511: recipe for target 'check-recursive' failed
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory '/home/aleksander/Development/foss/ModemManager/ModemManager-1.5.0/_build/sub'
Makefile:716: recipe for target 'distcheck' failed
make: *** [distcheck] Error 1

Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

The issue comes because intltool lists files within srcdir; and if builddir != srcdir, the files to ignore will be inside a subdirectory. The following patch makes an additional try to remove more ignored files, only when builddir is inside srcdir, by generating a path on top of builddir for each file to be generated. Horrible perl, so can very likely get improved in a better way... (but works for me for now) :/

Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

Wait, previous patch is wrong, as intltool already hardcodes the removal of the "_build" prefix.

Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

A quick and dirty solution (until next automake update) is likely to just add the new target prefix hardcoded, as done with "_build":

    foreach (@buf_allfiles_sorted)
    {
        my $dummy = $_;
        my $srcdir = $SRCDIR;

        $srcdir =~ s#^../##;
        $dummy =~ s#^$srcdir/../##;
        $dummy =~ s#^$srcdir/##;
        $dummy =~ s#_build/##;
++ $dummy =~ s#_build/sub/##;

Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

The patch will try to guess which is the subdir of srcdir in which builddir was set (e.g. when doing make distcheck). If there is such subdir (i.e. if builddir is within subdir) we'll filter out the subdir prefix of all files.

Before, make distcheck was handled just by removing the "_build" hardcoded prefix, now it's guessed in runtime (so should also apply when not doing distcheck and doing a normal srcdir != buiddir build).

dobey (dobey)
Changed in intltool:
assignee: nobody → Rodney Dawes (dobey)
Revision history for this message
Carlos Garnacho (carlosg) wrote :

Any progress here? AFAICT this still happens, and quite a few projects just sticked sub/... files to POTFILES.skip, which is not too ideal.

Revision history for this message
fujiwara (takao-fujiwara) wrote :

I also hope the 'sub' dir issue is fixed asap.

I'm fine to add just '$dummy =~ s#_build/sub/##;' in FindLeftoutFiles().

Revision history for this message
Vlad Orlov (monsta) wrote :

Last update in the intltool code repo was on 2015-03-09. A year ago. Guess it's unmaintained upstream. Funny that upstream is right here on LP as well...

tags: added: wily xenial
Vlad Orlov (monsta)
tags: added: patch
Revision history for this message
Aleksander Morgado (aleksander-m) wrote :
dino99 (9d9)
Changed in intltool:
assignee: Rodney Dawes (dobey) → nobody
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Vlad Orlov (monsta)
Changed in intltool:
status: New → Confirmed
Changed in intltool (Ubuntu):
status: New → Confirmed
Changed in intltool (Ubuntu):
importance: Undecided → High
Revision history for this message
Vlad Orlov (monsta) wrote :

Any progress on this for Xenial?

Vlad Orlov (monsta)
tags: added: yakkety zesty
Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

"WTF broke my distcheck!", then apply it to N projects using intltool, sum the amount of hours wasted by everyone applying workarounds in their own projects...

I have to admit this is very very sad.

A bug like this one, affecting every project using intltool, with one suggested patch even, still open 2 years later. Of course this is no longer critical because *every* project using intltool ended up patching their own POTFILES.skip... how very sad.

Let's see each other again here when automake updates again the default builddir during dist.

Revision history for this message
Vlad Orlov (monsta) wrote :

By some coincidence, I just sent Aleksander's patch to Debian...

Revision history for this message
Vlad Orlov (monsta) wrote :

Ok, here's the debdiff made from Aleksander's patch. Maybe we'll be more lucky with Ubuntu Sponsors Team...

tags: added: artful
removed: wily yakkety
Revision history for this message
Aleksander Morgado (aleksander-m) wrote :
Revision history for this message
Vlad Orlov (monsta) wrote :

Well, intltool is still needed for various projects which didn't migrate to gettext yet.

Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

Yeah I know, it's just not my problem any more. Everyone should migrate, gettext is at least maintained.

Revision history for this message
Vlad Orlov (monsta) wrote :
Revision history for this message
Vlad Orlov (monsta) wrote :
Revision history for this message
Vlad Orlov (monsta) wrote :
tags: added: bionic
Vlad Orlov (monsta)
description: updated
Revision history for this message
Vlad Orlov (monsta) wrote :

Is anything else needed to get the fix into Bionic before it's released in April?

Revision history for this message
Simon Quigley (tsimonq2) wrote :

Uploaded to Bionic, sorry for the delay!

Changed in intltool (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Vlad Orlov (monsta) wrote :

Thanks :)

Revision history for this message
Vlad Orlov (monsta) wrote :

Not sure if the upstream entry should be wontfix or not, it seems unmaintained.

Revision history for this message
Gianfranco Costamagna (costamagnagianfranco) wrote :

unsubscribing sponsors, subscribing release :)

Changed in intltool (Ubuntu Xenial):
assignee: nobody → LocutusOfBorg (costamagnagianfranco)
Changed in intltool (Ubuntu Artful):
assignee: nobody → LocutusOfBorg (costamagnagianfranco)
Changed in intltool (Ubuntu Xenial):
status: New → Fix Committed
Changed in intltool (Ubuntu Artful):
status: New → Fix Committed
summary: - intltool confused by separate build-dir
+ [SRU] intltool confused by separate build-dir
Vlad Orlov (monsta)
tags: removed: zesty
Revision history for this message
Aleksander Morgado (aleksander-m) wrote :

@monsta AFAIK, this bug report is the "upstream" bug report as well. As long as you're not the upstream intltool maintainer, please don't set it as wontfix. Even if it does seem unmaintained...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intltool - 0.51.0-5ubuntu1

---------------
intltool (0.51.0-5ubuntu1) bionic; urgency=medium

  * no-guess-builddir.patch: new patch by Aleksander Morgado.
    Fixes build of various projects when builddir is not the same
    as srcdir (e.g. when running make distcheck). (LP: #1117944)

 -- Vlad Orlov <email address hidden> Mon, 04 Dec 2017 17:21:57 +0300

Changed in intltool (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Vlad Orlov (monsta) wrote :

Yes, I meant the upstream entry here. I was just wondering, wasn't going to mark it (I don't actually have the rights for that).

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello zbyszek, or anyone else affected,

Accepted intltool into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/intltool/0.51.0-4ubuntu1.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed verification-needed-artful
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello zbyszek, or anyone else affected,

Accepted intltool into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/intltool/0.51.0-2ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed-xenial
Revision history for this message
Vlad Orlov (monsta) wrote :

Testing in Artful...

- upgraded intltool to 0.51.0-4ubuntu1.17.04.1 from artful-proposed
- repeated the tarball making process for mate-desktop package, exactly
  as described above in the Test Case
- the process completed successfully, the distcheck didn't break

Good to go.

tags: added: verification-done-artful
removed: verification-needed-artful
Revision history for this message
Vlad Orlov (monsta) wrote :

Testing in Xenial...

- upgraded intltool to 0.51.0-2ubuntu1.16.04.1 from xenial-proposed
- repeated the same tarball making process for mate-desktop package,
  the only difference being the git branch (1.12 instead of master)
- the process completed successfully, the distcheck didn't break

Good to go in Xenial as well.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intltool - 0.51.0-4ubuntu1.17.04.1

---------------
intltool (0.51.0-4ubuntu1.17.04.1) artful; urgency=medium

  * no-guess-builddir.patch: new patch by Aleksander Morgado.
    Fixes build of various projects when builddir is not the same
    as srcdir (e.g. when running make distcheck). (LP: #1117944)

 -- Vlad Orlov <email address hidden> Mon, 11 Sep 2017 17:56:28 +0300

Changed in intltool (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for intltool has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intltool - 0.51.0-2ubuntu1.16.04.1

---------------
intltool (0.51.0-2ubuntu1.16.04.1) xenial; urgency=medium

  * no-guess-builddir.patch: new patch by Aleksander Morgado.
    Fixes build of various projects when builddir is not the same
    as srcdir (e.g. when running make distcheck). (LP: #1117944)

 -- Vlad Orlov <email address hidden> Sat, 23 Sep 2017 14:01:01 +0300

Changed in intltool (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Vlad Orlov (monsta) wrote :

Ok... what to do about upstream now?

Revision history for this message
Gianfranco Costamagna (costamagnagianfranco) wrote :

@Vlad, just send them a mail or open a bug report

Norbert (nrbrtx)
tags: removed: artful verification-done-artful
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.