Cannot set pbuilder build locale

[Rob Savoury]

The build locale cannot be set with pbuilder and always defaults to "C" (for both LANG and LC_ALL on Launchpad builds) rather than to "C.UTF-8" as used by sbuild for all official Debian and Ubuntu builds. This causes FTBFS of various packages that depend on the correct locale being set on the build system, often noticeable with the test suites of Python packages for instance.

The attached patch changes only five lines of code (pbuilder-buildpackage and pbuilderrc) to allow easy setting of the pbuilder build locale. Based on local testing the patch works as intended, in that packages depending on the correct locale which FTBFS using unpatched pbuilder (yet build successfully with sbuild on Launchpad) now also build successfully with the local patched pbuilder. Please consider applying the patch.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "allow-setting-pbuilder-build-locale.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Jessica Clarke (jrtc27)]

I don't know about Ubuntu, but FTBFS under a common locale, which would include C, is regarded as a release-critical bug, so there is no "correct" locale, just the one that happens to be standardised across the Debian and Ubuntu buildds (otherwise, would "FTBFS in pbuilder" be a bug in pbuilder? because that's not the "correct" chroot-based build tool).

I don't know if this is the right way to fix it though. I feel like a more general bash associative array in order to set arbitrary environment variables might be the better way to do it, rather than specifically for LANG and LC_ALL.

We probably also should just change the locale in pbuilder-buildpackage to C.UTF-8; we might have some unicode-trashing bugs lurking in pbuilder itself otherwise.

[Mattia Rizzolo (mapreri)]

FWIW, one way to set LC_ALL in pbuilder right now is to set that in a config file that is then loaded from the cli with `--configfile`; that loading is done at a point that makes it possible to overwrite the current default.

[Rob Savoury (savoury1)]

So any use of a config file does not actually allowing setting LANG/LC_ALL for the dpkg-buildpackage invocation. Already in use on my personal build system is a detailed $HOME/.pbuilderrc config file and adding LANG/LC_ALL exports to this file did not affect the used locales for the actual build. When looking at the pbuilder-buildpackage script LANG/LC_ALL are both explicitly exported as "C" at the beginning of this main build script. This clearly makes any attempts to set them for the actual dpkg-buildpackage invocation fail.

Based on the above that is why the solution of adding two new variables BUILD_LANG and BUILD_LC_ALL was chosen, as these do not then conflict with the explicit exports in pbuilder-buildpackage. Given that I have not had time to debug all implications of changing those explicit exports in the main build script (and what changing them might potentially break) adding two new distinctly named variables was an easy and workable solution.

Also, I wouldn't really call this a "wishlist" bug due the fact that without addressing the issue of the locale exports in the pbuilder package one way or another it makes it impossible to actually use the same locale as for all official sbuild Debian and Ubuntu builds. This seems to be a serious limitation to me actually, as I've stated in the patch preamble.

Perhaps Jessica's suggestion of simply changing the two explicit exports in pbuilder-buildpackage to the same as used for official builds (ie. to "C.UTF-8") is a good one, but that would obviously require debugging the whole script to prevent possible breakage. In any case, the patched version that I've uploaded to my Launchpad build-tools PPA (ppa:savoury1/build-tools) works exactly as intended for anyone who actually does want a pbuilder that is capable of setting the locale for the actual build process.

[Jessica Clarke (jrtc27)]

pbuilder-buildpackage sets LANG and LC_ALL *before* sourcing pbuilder-checkparams, which is what sources the config files, so config files definitely override that. However, pbuilder-buildpackage then sources pbuilder-buildpackage-funcs, which itself sources pbuilder-modules, which then sets LANG/LC_ALL to C again. That's the actual reason it doesn't currently work.

[Jessica Clarke (jrtc27)]

And re being "a serious limitation", I agree it's less than ideal, but it should not be an issue for any policy-conforming package, for which not building under the non-UTF-8 C locale is RC-buggy.

[Mattia Rizzolo (mapreri)]

I'm quite sure setting it in a conffile loaded by --configfile works. It
doesn't in the standard /etc/pbuilderrc and .pbuilderrc.

On Sun, 17 Oct 2021, 7:51 pm Jessica Clarke, <email address hidden>
wrote:

> And re being "a serious limitation", I agree it's less than ideal, but
> it should not be an issue for any policy-conforming package, for which
> not building under the non-UTF-8 C locale is RC-buggy.
>
> --
> You received this bug notification because you are subscribed to
> pbuilder in Ubuntu.
> Matching subscriptions: pbuilder
> https://bugs.launchpad.net/bugs/1947424
>
> Title:
> Cannot set pbuilder build locale
>
> Status in pbuilder package in Ubuntu:
> New
>
> Bug description:
> The build locale cannot be set with pbuilder and always defaults to
> "C" (for both LANG and LC_ALL on Launchpad builds) rather than to
> "C.UTF-8" as used by sbuild for all official Debian and Ubuntu builds.
> This causes FTBFS of various packages that depend on the correct
> locale being set on the build system, often noticeable with the test
> suites of Python packages for instance.
>
> The attached patch changes only five lines of code (pbuilder-
> buildpackage and pbuilderrc) to allow easy setting of the pbuilder
> build locale. Based on local testing the patch works as intended, in
> that packages depending on the correct locale which FTBFS using
> unpatched pbuilder (yet build successfully with sbuild on Launchpad)
> now also build successfully with the local patched pbuilder. Please
> consider applying the patch.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/pbuilder/+bug/1947424/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: distribution=ubuntu; sourcepackage=pbuilder;
> component=universe; status=New; importance=Wishlist; assignee=None;
> Launchpad-Bug-Tags: patch
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: crichton jrtc27 mapreri savoury1
> Launchpad-Bug-Reporter: Rob Savoury (savoury1)
> Launchpad-Bug-Modifier: Jessica Clarke (jrtc27)
> Launchpad-Message-Rationale: Subscriber (pbuilder in Ubuntu)
> Launchpad-Message-For: mapreri
> Launchpad-Subscription: pbuilder
>
>

[Rob Savoury (savoury1)]

OK, so it DOESN'T work for me when using a custom configfile. Putting "export LC_ALL=C.UTF-8" into that custom config file and starting the pbuilder build results in the same "LC_ALL=C" still being set, although other LC_* env vars are now set to "C.UTF-8". So although it might be INTENDED to work by using the custom configfile, I'm still seeing LC_ALL=C regardless.

Also, the point is not that a "policy-conforming" package doesn't build, it's that in the real world various real builds fail due the incorrect locale. So in a "perfect" world then perhaps it wouldn't matter that pbuilder is using a DIFFERENT locale than all official sbuild Debian and Ubuntu builds. But in the actual real world it does matter, and therefore I still consider it a serious bug and limitation.

In any case, thank you all for your feedback, and have a nice day!

[andrew bezella (abezella)]

i'm going to add a "me too!" and a real world example. i needed to apply a patch to ansible-core which is in the list of "Packages in bookworm/amd64 which failed to build from source,"[1] despite it being a fairly prominent piece of software. it turns out this is due to a locale issue:
ERROR: Ansible requires the locale encoding to be UTF-8; Detected None.

i can confirm that using `pdebuild --configfile ~/.pbuilderrc-locale` with the contents
export LC_ALL=C.UTF-8
export LANG=C.UTF-8

was not sufficient to build the package. as noted above, `pbuilder-modules` eventually has the final say and resets them to "C". eventually, i gave up trying to find a "right" way to do it and updated the last lines of `/usr/lib/pbuilder/pbuilder-modules` to get the build done.

1. https://tests.reproducible-builds.org/debian/bookworm/amd64/index_FTBFS.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "allow-setting-pbuilder-build-locale.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[andrew bezella (abezella)]

ftr the debian maintainer of ansible-core patched its debian/rules to include:
export LC_ALL=C.UTF-8

which seems like an easy/clean solution to one class of problem.